A little SAP HANA DBA note on too many trace files…

+++ update 7.10.2015 – SPS 10

Since this blog was originally published some time and many SAP HANA revisions passed and there is meanwhile a supported easy way to do housekeeping around trace files.

There is the  ALTER SYSTEM CLEAR TRACES РSAP HANA SQL and System Views Reference РSAP Library command, that allows the clearing/deletion of trace files from the SQL command.

In addition to that, SAP HANA Studio allows to get rid of old traces via the UI:

And there is more: SAP HANA now automatically compresses and archives old trace files and you can also trigger that manually (mark the trace files you want to archive, right-click, select ‘compress’).

This means you probably won’t be in need of the technique presented in this blog. ūüėČ

Anyhow, for reference it’ll stay up here.

+++ update 7.10.2015 – SPS 10

If you have been running SAP HANA for a year or so, spanning multiple revisions, eventually overcoming bugs you faced in the past, you will invariably end up with an alert similar to this:

“There are currently 1033 diagnosis files.

This might indicate an issue with tracefile rotation, a high number of crashes or another issue. Please check the diagnosis files.”

As a matter of fact, SAP HANA is currently not particularly great at managing all the trace files it creates.

(And don’t get me started on the actual contents of those files…).

If things go really bad you may even end up finding error messages like the following in the indexserver trace files:

[…]

[89705]{0}[0] 2013-04-23 11:48:39.425976 e Basis        TraceSegment.cpp(00255) :

Exception while trace file compression:

exception  1: no.2120027  (Basis/Diagnose/impl/TraceSegment.cpp:238)

    Trace file compression finished with error: 18446744073709551613

exception throw location:

1: 0x00007fe11c8f5d80 in Diagnose::TraceSegmentCompressorThread::run(void*&)+0x13f0 at exception.hpp:313 (libhdbbasis.so)

2: 0x00007fe11c951e92 in Execution::Thread::staticMainImp(void**)+0x6b0 at Thread.cpp:457 (libhdbbasis.so)

3: 0x00007fe11c95208d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:528 (libhdbbasis.so)

[…]

All too bad, I know, but a quick and easy workaround for this is this:

  1. Create another folder, say “old_traces”
  2. Move all files older than, say 10 days (or whenever you last upgraded to the most recent revision), to this “old_traces” folder.

Sounds good? Great, then all you got to do is to follow these steps:

1. Logon to the SAP HANA server as SID<ADM>.

Be aware that in a scale out scenario, you will have to logon to each and every host and perform the following steps there.

2. Navigate to the trace file folder

For that, just use the predefined shell command alias ‘cdtrace

hansrv123:/usr/sap/HAN/HDB00> cdtrace

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace>

3. Create the “old_traces” folder

Note that in this example I simply create the “old_traces” folder as a sub-folder of the actual trace folder.

You might and probably should create it at a different location, to release the storage space in the trace file folder.

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace> mkdir old_traces

4. Move old trace files into the new folder:

This can take some time…

hansrv123:/usr/sap/HAN/HDB00/hansrv123/trace>find . -type f -mtime +10 -print | xargs -I {} mv {} old_traces/

If you find that you don’t actually need the trace files any more, you may just go on and delete this folder later on.

In general, it usually doesn’t make sense to keep the trace files generated by older revisions.

Exceptions to this general approach could be, that you faced bugs that you still wait to see fixed or to compare messages from normal/baseline system operations (think of time required for restart, message output during restart/shutdown/backup…).

There you have it – now you know! ūüôā

Cheers,
Lars

Back in the tar pit?

I’ve now been involved with a couple of SAP HANA projects and I am left astounded.

Not only by the impressing performance and flexibility our product delivers for certain use cases.

I’m much more astounded by the expected level of software maturity of the final project implementations.

Yes, I’m not talking about SAP HANA as a DBMS platform but of the systems created on top of it.

Barely any of the projects included specific plans to design a maintainable system.

A system that can be meaningfully monitored and eventually debugged.

Documentation of SQLScript code, modelled views and table design pretty much never exists.

Why did the designer of this CalcView decide to use plain SQL instead of CE_-functions?

What’s the rationale behind choosing this specific data type for that column?

How come this object is owned by user SAPDEV while the remaining ones all belong to MYAPPSCHEMA?

Answering this kind of questions requires having the actual developer still at avail.

Testing of views and procedures is done manually if done at all.

Automatic testing with protocols?

Not existent.

Test data sets to verify functional correctness (at least for given use cases)?

Never seen in any project.

Static and/or dynamic code analysis (How deep is your deepest function call stack you say? How many parameters you need to pass on average? What breaks if I change this part of your system?)?

Unheard of in SAP HANA circles.

This list can be extended with more and more points, but really irritating is the fact that apparently our paying customers are actually not asking hard enough for these things.

As it seems to me, they simply expect all this to be implicitly given. It’s an SAP system after all, isn’t it?

Fair enough or is it?

It seems the immaturity of SAP HANA as a platform is taken as a general excuse to throw overboard the preceding decades of software engineering and craftsmanship. (You might pick the label you want to put on professional, reliable and reproducible work here yourself.)

Implementation teams happily and knowingly produce code that cannot be maintained efficiently.

Of course, they do, as it will be their follow-up contract to fix and re-implement functionality.

Should the question come up, why the original code doesn’t do it anymore, it’s too easy to finger point at the lack of features and at instabilities of the development platform.

And is it just me or is the productivity of SAP HANA developers really that much smaller than that of their ABAP, JAVA or C# counterparts?

All this might work as long as the actual costs for the implementation aren’t bigger than the immense bill created by hardware and license costs.

Typically this means, round 1 of the implementation can fly under the radar of system quality, often leaving behind what could only be described as a mess.

No doubt, SAP HANA projects still are using a new platform. Many of the inherited best practices are questioned and some things needed to be changed.

But let’s not forget that at its very core, SAP HANA is a relational DBMS that exposes SQL as the main development API.

Now, how come that systems, that are built against an API that is 30+ years old, are implemented on such an unprofessional level?

Seriously, did just the fact that your IDE of choice doesn’t directly integrate with your version control system stop you from actually using it?

Did you consider using a SCRUM/agile approach with any other development environment that doesn’t allow for automatic testing?

I agree with most of the single feature demands seen here in SCN or communicated by customers and partners.

But I can’t agree with the lack of demand for a better system development practice.

And I certainly can’t agree with an approach where the goal to minimize system response time outweigh aspects like functional correctness and maintainability by several orders of magnitude.

The question is: why can you?

SPS 6 ante portas ‚Äď new features in SAP HANA Studio revision 55

Just recently our development team released revision 55 of SAP HANA Studio and I was very pleased to see how many improvements had been included this time.

Here’s a quick rundown of my favorites in no particular order:

1. Improved crash dump file viewer

If you ever had to review a crash dump or run time dump file in SAP HANA Studio you will have noticed, that the first things you have to do are always the same.

You open the file by double clicking on it in the trace file list.

Then you have to change the default selection of “Show End of File” to “Show Entire File” and finally you have to scroll up to the table of contents section, from where you start to review the file.

Typically one would then want to navigate to a specific section of the crash dump file, say the [CRASH_STACK], and so either the scroll bar of the text area is used or the text search feature is put to use.

All in all not really user friendly and a bit ‘geeky’ (although this maybe doesn’t bother the folks that typically would try to make sense of the crash dump files anyway…).

But no more!

Behold the new shiny crash dump file viewer:

The file automatically opens in “Entire File” mode and the text window shows the actual start of the file, thereby showing the table of contents.

And, tataaa, by pressing the [CTRL] key, the table of contents entries are turned into clickable hyperlinks that allow for a quick navigation within the crash dump file.

Nice!

2. Usage of Eclipse Platform 4.2.1 “Juno”

As SAP HANA Studio is based on the Eclipse Platform, it also gains from the advancements that this platform performs.

One example is the availability of OS dependent UI themes, like “Windows 7” or “Windows XP” that make the Eclipse platform look more like a OS platform native application.

A sample for the, sometimes slight, differences in the looks of the UI can be seen just here:

To try out the different themes, navigate to “Window” -> “Preferences” -> “General” -> “Appearance” and choose the settings in the dialogue shown below:

For more information the new Eclipse platform version, check the Eclipse website (here).

3. JDBC trace active indication

If you are like me, then you’ve been using the option to activate the JDBC trace in SAP HANA Studio quite extensively.

I use it for everything between actual trouble shooting to figuring out, what commands are sent to the database by SAP HANA studio, just to learn about what a specific feature really does behind the curtains.

If you haven’t done this yet but want to, no problem!

Simply right click on the DB connection icon in the navigator tree and select “Properties”. In the dialogue that is opened, select the “JDBC Trace” section and tick on the “[ ] Enable trace” option.

This feature allowed me to learn quite a lot about SAP HANA studio, but from time to time I forgot to turn it off again.

While this is not a problem per se, I’m one of those lazy guys that barely ever set up the maximum trace file size but always wonders why performance is not up to the scratch at times.

So I ended up with having super large JDBC trace files in my trace file folder (which are barely useful) and worse than necessary performance (due to the trace file writing).

But no more! (v2)

Behold the amazing JDBC active trace icon:

If I remember correctly, this one has been specifically built based on my wish, so a big THANK YOU goes out to the development team for implementing this!

With this icon, I’ll never forget an active JDBC trace anymore.

4. SAP HANA Studio provides more information about its current user

Let’s be honest: every one of us has been working in SAP HANA projects where users/developers/DBAs shared a common user account in SAP HANA.

Maybe something like “POWERDEV” or “SUPERADMIN” or whatever you fancy.

While we all know that this approach has its severe drawbacks, there’s probably no way to overcome this practice.

Unfortunately this makes it very difficult to assign a specific session/connection (e.g. one that holds locks on records or that runs a SQL command which allocates massive CPU or memory resources) to an actual user.

This kind of problem has long been solved though, as we faced it already with classic DBMS and SAP NetWeaver.

The solution is to use session context variables (I’m going to write about that more extensively in a separate blog post).

SAP HANA Studio now fills all of the four pre-defined session context variables:

KEY                     VALUE

APPLICATIONVERSION      1.0.55.201304261355   

APPLICATION             HDBStudio  

APPLICATIONSOURCE       createConnection(JDBCPlugin.java:274)

                        <-internalGetConnection(ConnectionResource.java:76)

                        <-getConnection(ConnectionResource.java:57)

                        <-getConnection(ConnectionResource.java:53)

                        <-run(SQLExecuteFormEditor.java:901)

                        <-run(ModalContext.java:121)

APPLICATIONUSER         I028297

With these variables set, it’s easy to identify which O/S users are actually using SAP HANA Studio and also which version of it.

APPLICATIONUSER in this example is set to my SAP network logon user name “I028297”.

The APPLICATIONSOURCE variable is set with a short JAVA method stack, which should allow finding out, which module of SAP HANA studio created this very connection.

The variable values can be reviewed, among other ways, in the “Administration Console” -> “Sessions”:

If you ask me, this is super useful!

5. Energy consumption reduction of the modeler editor views

Not sure if you ever stumbled over this, but personally, I was bugged a long time by the nasty CPU consumption of the modeler join editor views.

Even when you didn’t actually do something, the CPU was constantly humming along, which on my Laptop eventually led to the activation of the thermal fans (that’s how I initially found out about this).

After an internal support message and some revisions in between, the modeler editors now finally leave my laptop fans remain silent and the CPU consumption low.

I’m also under the impression that the whole graphical part of the modeler became more reactive to user interaction, but that might be just me…

Anyhow, well done dear developers!

Thanks once again for making SAP HANA Studio a bit more usable.

Hope you like the enhancements as much as I do.

Cheers,

Lars