Jul 18

SMS alerting via AQL

By tonvoon Opsview, Perl Comments Off

We came across AQL by accident. They came to us because they were interested in Opsview and we looked into what their company was about. They provide SMS messaging services: you buy a prepaid amount of credits and then you can send SMS text messages via their website in a variety of ways.

systempreferences.png

Our sales director thought it would be a good idea to integrate their service with Opsview. We agreed and thought there was a nice synergy about it.

So we’ve now integrated AQL’s messaging through our UI. In the upcoming 2.8 release, there’s a new screen: System Preferences. Here you can sign up at AQL and then enter the username and password. We even give you a little Check credits AJAX button for you to test your connectivity.

mobilenumber.png

Then on the Contacts screen, you enter in your mobile phone for sms number (with javascript validation so that it is in the correct format) and you can even send a test SMS to make sure this works correctly.

Finally, when Nagios is ready to alert, we send the notification via the SMS instead of email or RSS. Simple!

Actually, it was quite hard. We just like to make it look easy.

To communicate with AQL’s servers, there are various methods: HTTP/HTTPS, XMPP, SOAP and a few others that made my eyes water. We just wanted a nicely encapsulated module to send a message.

And we found one on CPAN. SMS::AQL is a perl module written by David Precious. It works over HTTP and worked a treat. However, we initially worked with version 0.02 and the tests there were failing because it was trying to contact AQL’s servers to do testing. This caused us some problems in our automated perl install.

So we set to work enhancing the module. First thing was to update the tests. Using Test::Mockobject, we were able to reply to SMS::AQL’s HTTP calls as if they were being returned from AQL’s servers. This allowed some really intensive testing. Using Devel::Cover, we got a 91% coverage in our testing! We found lots of inconsistencies in the API, which we fixed as well. Finally, we cleaned up the messages so there is a single lookup table now.

The guys at AQL have been very helpful in providing us with technical information. And David Precious has updated the perl module with our changes. And he’s written a blog post too!

It’s a symbiotic way of working – we didn’t start from scratch working on an interface with AQL’s systems, but we’ve managed to contribute back to existing code and move it up another level. Everybody wins! (Well, except for other monitoring system companies that want to be international conglomerates.) So now everyone can use the CPAN module to get SMS alerting.

But if you want a quick way of sending alerts, you can download our script here. This is the script that will be distributed with the 2.8 release soon. Just add that onto your server and put a check command entry into Nagios like:

define command {
command_name service-notify-by-sms
command_line /usr/local/nagios/bin/submit_sms_aql -u aql_username -p aql_password -n $CONTACTPAGER$ -t "$SERVICEDESC$ on $HOSTNAME$ is $SERVICESTATE$: $SERVICEOUTPUT$ ($SHORTDATETIME$)"
}

To be honest, I can’t remember all the associations with the contact definitions – check out the Nagios documentation to set it up. I just use Opsview because it makes Nagios easier to administer. And now, Opsview makes SMS alerting easier too.

Tagged with:
Apr 27

We have been asked by a customer if it is possible to change a check command for a service depending on the time of day.

Why would this be useful?

Well, if a server runs time critical processes during the day and slow running batch processes over night, how can a service check command take into account how it is supposed to report on CPU or memory usage without generating false alerts? Yes, you could write your own plugins to take account of the time and react accordingly for each check this needs to be for, but these would have to be installed on each host for each service, the wealth of plugins from http://www.nagiosexchange.org/ cannot easily be used, setting the system up takes longer, and it is all much harder to maintain.

Instead, we have made changes to the service stanza within the Nagios configuration files to include a “check_timeperiod_command <timeperiod>,<command>” entry:

define service {
host_name server1
service_description Free Widgets
check_command check_widget -w 40% -c 20%
check_timeperiod_command nonworkhours,check_widget -w 5% -c 2%
.....
}

You get the idea….

check_command provides the default check for the day. During the nonworkhours period, the alternative command and arguments are used instead.

This seems far too useful to the community to keep to ourselves, so we offer the patch for Nagios 2.8 here, for peer review and comments (all of which are very welcome).

And here is a patch for ndoutils 1.4b2 that goes with it.

Enjoy!

Update: Patches for Nagios 3.0.6 and NDOutils 1.4b7 are available

Tagged with:
Jan 26

We ran across a problem with NSCA 2.6 yesterday day. It turned out that running the nsca daemon in single mode only works for the first packet of data from send_nsca and hung for subsequent calls.

This was actually first discovered by Rudolf van der Leeden and it looks like it has been with us since April 2006 when NSCA 2.6 was first released, through to the current NSCA 2.7. We never picked it up until running it on a customer site which was tuned to use –single.

The fix is as Rudolf suggests – uncommenting the if statement that was removed. Our patch is here.

How do we know it works? Well, we’ve written a series of test scripts for NSCA.

We’ve always been a big fan of testing. We love using the Test Anything Protocol (TAP) in Perl. CPAN encourages you to write good tests to make sure your Perl modules run, which is why we know that modules we’re uploaded to CPAN continue to work while we’ve been updating them. And we’ve provided quite a few fixes to CPAN modules where the tests fail (and some just suggest that we have a broken version of perl).

Here’s the test scripts for NSCA. They are more like functional testing – it tests that the daemon can start up and accept messages and compares the output in the dummy nagios.cmd file with the sent data. Unit testing is a bit more tricky to do for C code – though libtap is being used for the Nagios Plugins.

To use the test scripts, drop it down to the top level of the NSCA directory after you’ve compiled NSCA and cd into nsca_tests. Run prove *.t. You will require several CPAN modules: Test::More, Class::Struct, Clone and Parallel::Forker, though most will be with your perl distribution.

There are 3 tests at the moment:

  • basic – just sends a few passive checks and makes sure that the nagios.cmd file receives them
  • multiple – runs the same as basic, but several times to check the daemon can handle multiple requests
  • simultaneous – runs lots of send_nscas at the same time (well, nearly). Uses Parallel::Forker to setup all the sends then executes them all at once. Expect about 200 extra processes to hit your server!

You’ll find that multiple and simultaneous tests fails with the stock NSCA 2.6 and 2.7. But when the patch is applied, all the tests work.

The tests can obviously be extended, but this is a start and covers this basic functionality.

We hope Ethan will look into adding this to the NSCA distribution.

We’re upset that something like this got to one of our customers, but we’re more upset with ourselves for not catching this much earlier. This should be a good step towards better QA of future NSCA releases.

Update: Ethan has updated NSCA to 2.7.1 to fix this problem. And the tests are included as well!

Tagged with:
Jan 23

There has been a new update to NDOUtils to 1.4b2 recently and we thought we’d share our latest patches here so that they can be evaluated upstream.

We’ve always argued that it is best to be as close to the released code as possible – we don’t want the expense of maintaining a fork, so it’s in our interests to inform everyone about our changes. And since NDOUtils is gearing up for the 1.4 release, now is a good time to publicise them.

Our course, the link to our most stable code is updated daily, so the list below will not be accurate over time, but we’ve also uploaded the patches onto this blog server, so you can still reference them here. All patches will apply cleanly onto NDOUtils 1.4b2.

ndoutils_issue_commands.patch

This is the include header problem because we’ve changed the data structure for a contact. Long term, it is best if Nagios splits the include files out of NDOUtils and let it be installed by Nagios, but this is probably outside of Ethan’s radar right now.

ndoutils_daemonize.patch

We found that the ndo2db process wasn’t closing stdout, which meant the attaching terminal could not close. It looks like it should be set, but was commented out for debugging purposes. We uncomment those lines.

ndoutils_debug.patch

It looks like some memory debug is switched on by default. We switch them off here.

ndoutils_memory.patch

And the ifdef doesn’t actually switch them off – we correct that too. (We’re too lazy to combine these last two patches together!)

ndoutils_configure.patch

The configure script wasn’t respecting the –with-mysql-inc option correctly. We also test for the compress lib, which gave us problems on Mac OS X.

ndoutils_notification_level.patch

This is required to support our use of simple escalations. Again, a separate location for the include files for Nagios would remove the need for this.

ndoutils_clear_tables_on_reload.patch

This is the biggie. If the configuration for Nagios has changed and a reload requested, the ndo_object table do not reflect the new configuration. We found that the ndo_objects table is only updated on a restart, not a reload. This caused problems for status views that use the database because the new hosts and services weren’t there. This fixes that problem.

We also found that the active flag wasn’t correctly set to inactive when the configuration was dumped. Once we fixed that, we found that hosts and services in ndo_objects were marked inactive, when they should be active. This has also been fixed, along with a SQL typo.

Update: we’ve discovered that the configdumpstart routine gets called twice – once with the original data, and once with retained data. Looking at the ndomod data stream, it looks like the configdumpstart is sent with a huge set of data, then another configdumpstart with more data. The patch above has been re-worked so that the table clearing only occurs once, before the original data is sent through. This does beg the question of what is the difference between the original and the retained data – if there was a table clear happening between the original and retained data yet all data was there, why send all the original data?

Also, we found a bug where configdumpend was not being called. It turned out to be a missing break in the case statement. This is included in the patch above too.

DEFAULT CHARSET mismatches

We also run a perl script when the NDOUtils distribution is unpacked. We strip out all the DEFAULT CHARSET=ascii statements in mysql.src. This is because if the server has a different charset, you can get some collation errors in mysql. We think it is better to remove these altogether and leave the charset to be set by the mysql database. The script is:


perl -pi -e 's/DEFAULT CHARSET=ascii //' db/mysql.src

ndoutils_upgradedb.pl

Upgrading database schemas are a terrible pain. NDOUtils includes scripts to update the database, but there’s a manual step required to work out which scripts to apply. We’ve written a perl script (requires DBI.pm) to apply the upgrade scripts automatically, as long as the filename convention is adhered to. There’s also a new table created, called nagios_database_version, which holds a single row with the version of the database schema for subsequent updates.

The copyright for this script can be claimed by Ethan if he chooses to include it in the NDOUtils distribution. Otherwise, you are free to use it and distribute it yourselves under the GPL, but the copyright is retained by Altinity.

Hopefully, these patches will get included into the new NDOUtils soon, as we move forward to the next generation of Nagios status viewers.

Update 2: Ethan has applied these changes to CVS, except for the notification_level patch as that is a bit more involved.

Tagged with:
Nov 02

The problem

For one customer, we had a major scaling issue with distributed monitoring and NSCA. The initial setup was one master, 5 slaves using send_nsca to send passive service check results back to the master. This is the standard setup, with the ocsp_command like the submit_check_result script.

But we started to see some bad figures in the Nagios performance. The average Check Latency was showing 9.5 seconds, which seemed far too long. On the master, we could see 50+ nsca daemon processes, though they didn’t appear to be doing anything.

The revelation

The revelation came when we looked on the slave. At any one time, there was only one send_nsca running! So even though the service checks were being run in parallel, it looked like ocsp_commands were being sent serially. This had to be our bottleneck.

The solution

So we wrote a script called send_nsca_cached to cache the passive check results. The idea is that the script will take the results as usual, but write to a cache file instead of running send_nsca. This cache file would hold a start time, so if the current result exceeded the start time + cache period, then send_nsca would be invoked and send all passive results at once.

We put the script on the slave and could see that the cache file would fill in spurts – 10 entries looked to be written within half a second, but then nothing for a few seconds. Nagios does some tricks to try and spread the service check load, but I wonder if the “traffic jam” of sending the uncached way was causing the services to be bunched up together.

When we checked again in an hour, the maximum Check Latency dropped to under 1 second and the master had only 9 nsca daemons. And I guess it is much better for network load as well to send a whole bunch of data at once, rather than a single message at a time.

The warnings

There had to be some bad points.

  1. This script is only for Nagios 2.0+ because of the use of environment variables
  2. We don’t support passive host checks. Not sure if this is a good or bad thing
  3. Do not use this if your slave is not busy. As send_nsca_cached needs to be invoked in order to send results, if your ocsp_command is only invoked once every minute, then the quickest you will get a batch result sent to the master is every minute, regardless of your cache time. So only use this script on a busy slave. You could use a cache time of 0 to be the same as sending immediately
  4. Don’t make the cache time too large. The results have no timestamps, so when Nagios on the master receives the results, it will process it as if the check happened just then. Also, if there is too much data being sent, you could fill the command pipe on the Nagios master
  5. On that point, make sure the master Nagios server has command_check_interval=-1 in nagios.cfg, so that the command pipe is read as quickly as possible. There are known limitations that if the pipe is filled, processes writing to the pipe will hang until more space is available

The future

That last point about the command pipe is being (partially) addressed in Nagios 3.0. Ethan has said at the Nagios Conference in Germany there will be a new external command called PROCESS_FILE, so the idea is that nsca can drop a file down on the master with a file containing passive check results and then only one command is put into the pipe, which will then process that entire batch.

The real solution to point (3) is to let the caching be done at Nagios, rather than externally, and that is also on the radar for Nagios 3.0. So there is lots to look forward to there. But if you want something now, check out our script. It’s not a perfect script because it’s hard coded in various places and you will need to customise the send_nsca command, but we hope it helps you regardless. Enjoy!

The end?

Not quite. At the Nagios Conference, Ethan was talking to two guys who were complaining that their distributed setup had huge slowdowns. I overheard and the symptoms looked exactly the same, so I gave them a copy of the script. Apparently it helped, but they had some lock ups in Nagios, which they think were attributed to our script – so caveat emptor. They have since reverted back to using the standard uncached mechanism.

We haven’t had any issues for our customers, so we’re interested in what you find. If you have a distributed environment with similar symptoms and you are thinking of using this script, please take a note of your Check Latency and the number of nsca daemons and add a comment to this blog with some before and after statistics. We’d love to know if this works elsewhere. Good luck!

Tagged with:
Oct 18

We had a customer problem where there were hundreds of NRPE processes on one of their monitored servers. It was quite bizarre because strace wouldn’t attach to the process. Lsof said there was an established connection from the Nagios server, but when we looked on that server, netstat said there was no such connection! I’ve never seen anything like that before!

Well, the customer’s network team said there was significant packet loss between the Nagios server and the NRPE box (obvious really, when all services to that host were complaining about SSL handshakes). Syslog showed lots of errors too:

Oct 17 10:41:46 host nrpe[2300]: Error: Could not complete SSL handshake. 5
Oct 17 10:42:26 host nrpe[2317]: Could not read request from client, bailing out...
Oct 17 10:42:26 host nrpe[2317]: INFO: SSL Socket Shutdown.

It looks to me like the SSL handshake was probably continuing to retry, but the connection must have been severed because it took too long. When our client tried to do an ssh onto the NRPE server, it was taking too long and he had to Cntrl-C to break out. We realised that NRPE should have some sort of timeout itself too.

So we’ve created a patch to NRPE 2.5.2. There is a new parameter in nrpe.cfg called connection_timeout. NRPE now sets an alarm just before handling a connection and then resets it before running the check command. It would have been best to have an alarm set over the entire session, but my_system sets an alarm handler too to make sure the command being executed does not exceed its timeout. This problem is probably SSL only, but the patch is over the connection regardless.

Testing on our customer’s servers, we found that check_nrpe returned CHECK_NRPE: Error – Could not complete SSL handshake as expected and the nrpe daemon then died gracefully when it exceeded the connection_timeout parameter.

It would be too ironic for a monitoring system to cause a box to die – although I hear BMC Patrol has that feature :)

Tagged with:
Sep 14

We had a customer that was interested in having multiple levels of escalation, so, for example, the 3rd notification was received by the line manager but the 5th notification would get to the head of department. If it got to the 10th – the CTO would come screaming!

Since Nagios has support for escalations, we worked on getting this functionality exposed in Opsview. However, the definitions got very complicated.

The point of Opsview is to simplify the configuration, but the service escalations in Nagios required an administrator to define data in hosts, services and contacts. In the background, we had to define different contact groups for escalation people, and this meant we had twice as many contact groups as we originally had.

And if we wanted extra levels of escalations, well, the configuration would exponentially explode!

It quickly got out of hand.

When we thought about it a bit more, all we really wanted was another filter on a contact level. Something like:

define contact {
contact_name	manager
...
notification_level	5
}

So the manager would only get notified from the 5th notification onwards.

This could be done in a notification script using the NAGIOS_NOTIFICATIONNUMBER environment variable, but then notifications.cgi would show an email to the manager, even though they didn’t actually get one. We believe this sort of logic should be in Nagios, not in an external script.

So here’s our patch. It applies to Nagios 2.5, but you need to apply the can_submit_commands patch first because the change is in the same area of code. There’s an associated ndoutils patch as well, if you use that.

To use, just add notification_level in the contact’s profile and you’re off! It defaults to 1 to keep current configurations happy.

Enjoy!

Warning: We’ve spotted a problem with this in our testing. If a service fails and no-one gets a notification because the contact filters fail, then the notification number reverts back to 1. So if a manager is set to get the 2nd notification, but there are no other contacts that get the 1st alert, the notification number will never get to 2. Thus no one will ever get an alert. This is serious. We plan on amending the preflight checks so that it fails unless each contact group has at least one contact with notifications enabled and a notification level of 1. In the meantime, if you use this feature, make sure an operations team has notification level of 1. Or, you could setup RSS alerts for every failure.

We’re not convinced this notification number should be decremented, but we’ll have that discussion in the mailing lists.

Tagged with:
Sep 11

We are starting to use Ndoutils, which is the first event broker for Nagios. The idea with the event broker modules is that the functionality of Nagios can be extended without the core code being changed. Ethan has released ndoutils which writes Nagios data to a mysql database.

We managed to get Ndoutils 1.3.1 to compile, but whenever Nagios started up, we kept getting SIGSEGV and Nagios would crash. Nagios.log would say:

[1158012772] Nagios 2.5 starting... (PID=21793)
[1158012772] LOG VERSION: 2.0
[1158012773] ndomod: NDOMOD 1.3.1 Copyright (c) 2005-2006 Ethan Galstad (nagios@nagios.org)
[1158012773] ndomod: Successfully connected to data sink.  0 queued items to flush.
[1158012773] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1158012773] Caught SIGSEGV, shutting down...

It took us a few hours to work out with lots of debugging lines in the ndoutils (printf statements + starting up Nagios manually without daemonizing), but we eventually found out that the Nagios 2 header files distributed with Ndoutils did not have our changes for can_submit_commands. The patch is here – obviously, only use this if you are using the can_submit_commands patch.

We’ve been speaking to Ethan because we think it is a good idea for Nagios to install the header files (maybe in /usr/local/nagios/include?), so then any local patches are done there, rather than trying to maintain multiple header files across different projects.

Hope this saves you a few hours!

Tagged with:
Jun 30

This is a summary of a question I posed on the nagios-users mailing list.

In a distributed environment, we want the slave Nagios servers to do the alerting. The Nagios documentation says that the master should do the notifications as this is the central point of control, but we think there are two major limitations:

  1. the slave is not autonomous – if the connection to the master goes, then no notifications are released
  2. with slaves in different countries and local operators, the paging of notifications shouldn’t come from a central server

However, we had a problem. Some forms of notifications should be run on the master, such as RSS (we will be releasing this separate addon next week) or helpdesk integration. Another limitation is that we only allow NSCA communication from the slaves to the master.

The mailing list came to the rescue. Thanks to all the people we got responses from.

Marc Powell suggested putting logic in the notification scripts on the slave to check if the master is up then forward to master, otherwise notify itself. But I discounted that because I’m opposed to putting “should I actually notify or not” type of logic in the notification scripts – there’s just extra code to support and this is what Nagios should be doing.

Patrick Morris has slaves set with notifications off and the slaves check whether the master is working ok. If this fails, then switch on notification on the slave. I like this idea, although it requires pager notifications on the master to be “distributed” so that it forwards requests to a slave for local dispatch. However, one possible problem was raised by Robert King in a thread called Forcing renotification of existing states, where switching on a slave’s global notifications will miss out on services that are already in a non-OK state at the time of the switchover.

After thinking about it some more, we decided to create multiple contacts per person. On the slaves, we have the usual contact called user, but with host/service-notification commands of email and pager (if desired). However, it got complicated on the master because the usual user contact had to have no notification options. We then created two extra contacts:

  • user/distprofile – for only master generated notifications
  • user/masterprofile – for the usual notifications on the master

It’s all a bit ugly. The big downside to this is that there are many more object definitions – for each contact group, we also needed to create corresponding /distprofile and /masterprofile ones too. However, since we generate the configuration, the pain is a one off.

Looking long term, we decided the solution is if Nagios implements some sort of contact profile, defined like this:

define contactprofile {
contact_name                  contact_name
contactgroups                 contactgroup_names
host_notification_options     [d,u,r,f,n]
service_notification_options  [w,u,c,r,f,n]
host_notification_commands    command
service_notification_commands command
}

The contact is left with static definitions like email and pager, while the contactprofile can be sliced and diced for specific host/services. A lovely feature of this is that you can define warnings to be sent as email, whereas criticals are paged.

But that’s some way into the future…

Tagged with:
Jun 07

Last time we looked at how to get SNMPtraps received into Nagios. This time we’ll show how Opsview handles the configuration of it.

Recall that the new design is:

  • SNMP packet received by snmptrapd
  • snmptrapd’s traphandle calls snmptrap2nagios
  • snmptrap2nagios, if applicable, will write to the Nagios command file

In Opsview, we use a web interface to configure the traps we are interested in. On this screen, we define the traps we expect to receive.
list_traps.png


Each trap has an alert level and a message. The message can use macros which are supported by our perl module SNMP::Trapinfo. You define them on this screen.

define_trap.png


If desired, you can deny this trap, so when the trap gets to snmptrap2nagios, it will be discarded. It is possible to deny the trap at snmptrapd, but we haven’t done that (as you normally have to be root to change snmptrapd’s configuration file. However, this is a worthwhile enhancement if there are lots of traps received).

But that’s not all! If a trap is allowed, you can then select to process it or ignore it at the host level. Here’s the configuration screen you get for defining your service check. This service check is then associated with a host.
defining_servicecheck.png

What is the defer option? Well, we thought there are 3 possible actions when a trap is received:

  1. You want to process it
  2. You want to ignore it
  3. You weren’t expecting it!

So defer means you haven’t said either way. This is the default for any new traps.

When this servicecheck has been linked to a host, Opsview will then configure Nagios with a service check called Interface which will accept traps linkUp and linkDown. The state of this servicecheck in Nagios will change dependant on the alert level defined for these traps.

So we have 2 levels of filtering:

  • globally deny a trap
  • ignore a trap on a host basis

What if, say, a Security trap arrives for a host that does not have the service associated with it? This is an exception, which needs manually intervention. Opsview has a table called snmptrapexceptions which stores all the traps that snmptrap2nagios hasn’t been told what to do with. When we thought about it, there were 5 distinct error conditions:

  • A trap was received with no valid trapname
  • A trap was received where the trapname was not fully translated – this usually means a MIB file has not been loaded
  • A trap was not recognised – it has not been defined
  • A trap was received, but it was not expected for this host (defer)
  • A trap was received for a host that is not defined to Nagios

We have a screen which shows all the exceptions and then gives operational options on what actions to take next.
exceptions_list.png


Notice there is a Promote Mib button. When we distribute Opsview, we put all our known MIBs into /usr/local/nagios/snmp/all. However, there is a penalty with loading unnecessary MIBs. So we configure snmpd to only load mibs in the default area and /usr/local/nagios/snmp/load.

When you click Promote Mib, we use a perl module called Net::Dev::MIBLoadOrder, which can tell you which MIB a specific OID belongs to. We then copy that MIB file into /usr/local/nagios/snmp/load and restart snmpd. This is one major administrative headache reduced!

Once you’ve redefined the actions you want, we tell Opsview to reprocess all the snmptrap exceptions based on the new rules (but no passive checks are submitted to Nagios). This will reduce the exceptions table so then an administrator would continue to set new rules until there are no exceptions left.

Astute readers may be wondering: “what happens if I receive a trap which is bad (linkDown) and then a good trap (linkUp) on the same service check”? The answer is that the bad trap will make the service go CRITICAL/WARNING, but the good trap will make the service OK. This means it may potentially get lost. We’ve made a decision to take this limitation, rather than use the is_volatile or stalking_options. However, we’re in discussions with Ethan to see if we can enhance Nagios to cope with this types of event.

Our aim is to make Opsview as easy to use, while we continue to improve Nagios and the general open source universe, through software or knowledge sharing. We hope this gives you an insight how you can get SNMPtraps working with Nagios.

Tagged with:
Nagios © 1999-2011 Nagios Enterprises LLC. Nagios, the Nagios logo, and Nagios graphics are the servicemarks,
trademarks, or registered trademarks owned by Nagios Enterprises, LLC. All Rights Reserved.
Opsview © 2008-2011 Opsera Ltd. Opsview, the Opsview Logo, and Opsview graphics are the
trademarks or registered trademarks owned by Opsera Limited. All Rights Reserved.
preload preload preload