How to take care of index corruptions in primary key indexes.

Primary key constraints in Oracle are enforced by using an index. The index does not need to be a unique index, but it has to be possible for the database to look into it and find out if a given key is already present or not.

This works fine, as long as the index structure is not corrupted. If it does, the standard recommendation to fix this was some procedure like this:

  • Stop your SAP-system to prevent the entry of duplicate keys (these are nasty to remove later on!)
  • Drop the index
  • Rebuild the index
  • Start the SAP again

There are several things that are not so nice about this:

  1. It requires downtime.
    We’re living in 2008 and still, the recreation of a secondary data structure can only be done in a downtime!? Not too nice.
  2. It requires the DBA to be handy with the Oracle Dictionary.
    He/She will have to find out the index-definition commands (DDL) before the index is dropped. Of course, it is no problem to do so, but you have to know how – and many do not. (You can use DBMS_METADATA.GET_DDL, the EXP/IMP tools or the Oracle dictionary directly, just to mention three of the options to get the DDL for the index).
  3. It involves the risk, that – although the SAP is shutdown – duplicates keys can be entered into the table.
    Ok, this is rather unlikely, but it is a potential risk – it is not bulletproof this procedure.

To get around this, I made a little research and found two possible ways out.

Solution #1

First, the nice and easy one, that now has found its way into the #365481 – Block corruptions note:

Use ALTER INDEX … REBUILD ONLINE.

It is as simple as that.

Why didn’t we use that earlier?
Because the Oracle documentation says that for the REBUILD of an index, the necessary data is taken from the existing Index segment and not from the table (they changed that for Oracle 11g! See http://download.oracle.com/docs/cd/B28359_01/server.111/b28274/data_acc.htm#i7530 “14.1.7 Re-creating Indexes” ).
So if the existing index is corrupt, we cannot use the REBUILD clause.

What the Oracle documentation does not mention is that the behaviour for a REBUILD ONLINE is quite different. When used with the ONLINE option the REBUILD accesses the table directly instead of the old index (an elaborate discussion on this can be found in Oracles Metalink: Note:278600.1)

With this knowledge, it is now possible to perform the Index rebuild while SAP is up and running. (Please do check the mentioned note before nevertheless. Even a REBUILD ONLINE has it’s “dangers”).

Solution #2

So, that was solution number one. It is easy, it is safe – you should use it if necessary. But actually, it was the second solution I found.

The first one was still based on the assumption that for an index rebuild the old index has to be read.

To explain my solution, I will make up a little example:

SQL> create table pktest (mandt varchar2(3) not null, name varchar2(40) not null, id integer);
Table created.
SQL> create unique index i_pktest on pktest (mandt, name);
Index created.
SQL> insert into pktest values ('100', 'Test1', 1);
1 row created.
SQL> insert into pktest values ('100', 'Test2', 2);
1 row created.
SQL> insert into pktest values ('100', 'Test3', 3);
1 row created.
SQL> insert into pktest values ('100', 'Test1', 3);
insert into pktest values ('100', 'Test1', 3)
*
ERROR at line 1:
ORA-00001: unique constraint (SAPR3.I_PKTEST) violated

==> as we see the unique index takes care of our primary key constraint.

Be aware that SAP does not create any primary key constraints as such on the database.

Instead, the key columns all get the NOT NULL constraint and a unique index is defined on the key columns. While this makes technically no difference, it is a semantic difference compared to a primary key constraint!

SQL> commit;
Commit complete.

==> Now let’s make the index broken…

SQL> alter index i_pktest unusable;
Index altered.
SQL> insert into pktest values ('100', 'Test1', 3);
insert into pktest values ('100', 'Test1', 3)
*
ERROR at line 1:
ORA-01502: index 'SAPR3.I_PKTEST' or partition of such index is
in unusable state
==> as we see it does not work anymore.

Unfortunately, with a real index corruption, the index is not automatically flagged UNUSABLE. Therefore, inserts/updates/deletes can happen as long as they do not require access to the corrupted blocks.

==> Here comes the solution…:

We create a new index that provides the same uniqueness constraint (that all combinations of „mandt” and „name” have to be unique. To do that we’ve to choose a different order for the columns in the index.

NOTE: this only works because the NOT NULL constraints are in place. Otherwise changing the column order in the index would not lead to an equivalent uniqueness check!

SQL> create unique index i_pktest2 on pktest (name, mandt);
Index created.

==> get rid of the broken index:

SQL> drop index i_pktest;
Index dropped.
SQL> insert into pktest values ('100', 'Test1', 3);
insert into pktest values ('100', 'Test1', 3)
*
ERROR at line 1:
ORA-00001: unique constraint (SAPR3.I_PKTEST2) violated

==> the new index/constraint works well

==> now we’ve all the time in the world to fix the broken index:

SQL> create unique index i_pktest on pktest (mandt, name);
Index created.

==> a little cleanup, et voilà!

SQL> drop index i_pktest2;
Index dropped.

The same pattern can be applied if really unique or primary constraints have been defined on the table. In these cases, it will be necessary to create additional constraint definition on the key columns as well.

Of course, this solution should not be used, if a REBUILD ONLINE is possible, but it is a working solution to the problem and it may happen that REBUILD ONLINE won’t work – then you can think about this approach.

Best regards,
Lars

Point-in-time-recovery considered harmful

When I first got in contact to the possibility to bring back a database to the state it had, when a backup was taken I was not too much impressed.
I though “Well, it’s like any other backup – copy the saved file back to the place I found it.”

Getting deeper into the tasks of a database developer (who had to take care of the database instance as well), I soon figured out that there is a much cooler feature available: point-in-time-recovery (PITR).

This feature allows to re-execute every change made to the database after the backup has been taken, so that it is possible to bring the database right back to any state it had between the time the backup has been taken and the moment where most recent change had been done.

One major opportunity of this feature that is broadly advertised by all database vendors: the ability to undo human mistakes – even if they are recognized hours or days after the mistakes happened.
Just recover the latest backup and ‘roll-forward’ just before the disastrous “DROP TABLE” was issued (for some reason this must be really the most frightening thing to the developers of the database documentation …).

So if this is that great – what is wrong with it?

The feature itself is a very nice thing that – applied correctly – brings many advantages.

However, the problem is that it is most often not applied correctly. Although the PITR guarantees to result in a transaction-consistent database, the use of it may lead to severe inconsistencies in your “data world”.

Usually database developers expect a database to be the only storage of information. When this assumption hold true, the PITR is completely nice. Unfortunately this assumption is always never true.

Synchronize with the real world

The information stored in the database represent facts we know about the reality. That could be orders, invoices, customer names and the like.
If we set back just the database to a state it had in previous point in time the facts in the real world are not sat back as well.

Even worse, the real world has no way to know that the database from which your company takes all information for its business processes is not “up-to-date” anymore.
If your real world supplier gets a second order for material he usually will not suspect that you made a mistake – he will fulfil the order and invoice you.

It is even worse – in today’s companies there is not a single database installation, there are tens, hundreds or thousands of production database running. Many of them are connected via interfaces to synchronize data and trigger the execution of processes based on this data. Now assume one central database to be set back to a point in time, let us say yesterday morning. All processes triggered and eventually finished since then, will be re-triggered.

Ask yourself, what would happen in your company if such a thing occurs.

Root cause

So what is the reason for all these problems?
Basically, the problems occur because there is no general synchronization between systems.
Systems cannot be synchronized well by timestamps as it is impossible to bring the exact same time to every system – at least the deviations will not be small enough.
Therefore, database systems rely on time-independent mechanisms to bring order to the sequence of changes made to the data.

In Oracle, this is the System-Change-Number, in MaxDB, it is called log-IO-number and there are similar concepts in the other DBMS as well. DBMS just keep counting the number of changes that had been done to the data. That way it’s possible to say: “Ok, I restore a database backup where the last change was change number 1000 and now, by using the redo-information, I roll-forward to change number 1200”.

Obviously, this change counting approach can only work within one system.
Other systems (databases, web-services, real world…) all have their own change counting in place (for the real world this would be the real time) – all of them completely separated from the other.

One way out

One way to come around this is to implement the change counting on a lower level of the IT-structure of your company: at the storage level.

If you keep all data ever stored in your company on a single storage facility (SAN, NAS, … ) than this storage-“thing” can do the change-counting to keep track of all changes made to the whole data-“world” of your company.
With a setup like this, it is of course possible to perform a PITR of your whole data-“world”.
Anyhow, even if now all data in your company is consistent again: the real world is still not set back in time. Your supplier still would deliver the order that has been sent a second time.

So, what can we do about it?

The easy answer is: Do not do point-in-time recoveries!
Do not take them as a valid option to get your data back quick and easy!

In most cases where a PITR has been done, the major effort had to be invested into making the database consistent again with other systems and with the real world afterwards.

Even the often taken approach to restore “just” one table from a database backup in a different instance up to a certain point in time, then copy the table to the target database needs huge effort to result in a consistent database again. For SAP systems, it is yet worse as referential constraints are not implemented at database level but only by the SAP work processes.
Therefore after such a partly PITR of the database one even has to run check reports (if such ones exist for the table in question) to be sure the database is consistent again.

In my view, PITR is not a feature for production databases or databases that are connected to any other system or that have any real relevance to the real world. So which one could that be?

Test systems, Training-Systems and  Q/A-Systems are the only systems that could bear a PITR.

All other systems, especially production- and development-systems should never be exposed to a PITR.
Obvious to see – these are also the systems where nobody should ever be able to manually drop a table or delete all data of a table.

But we need this feature…

Now – what should you do if a user accidentally deletes important data?
What if the user does the wrong change to the wrong data entry (e.g. to invoicing the wrong customer)?

The answer is “It depends on what you would do in the real world.” If you send an invoice to the wrong customer, you usually would try to cancel it in your system and try to prevent it from delivery.
If the wrong invoice is delivered, one would send a mail to say “Sorry, this was a mistake, please ignore the invoice. Thanks!”
Therefore, your application should give the same options to you.

If it is possible that a user makes a mistake that could be undone like any action in WORD – the application should implement UNDO functionality for that (for example keep versions of specific data).

The punch line is: complex UNDO-functions have to be implemented at the level where the changes happened – not on any low-level like the database.

So better forget or banish the PITR option from your DBA toolset.
It is much more dangerous than useful.

Debugging the MaxDB Kernel – How to get a Call stack the easy way

A few months ago, I described how to set up and configure the Windows Debugger in order to be able to generate a call stack for a hanging MaxDB thread.

As this was technically interesting (at least I hope so), it is by far not the simplest way to get a call stack for a hanging task.
Moreover, due to the internal tasking model of MaxDB you cannot specifically point to a usertask (that is the session of a user) and debug it.
All you can do is to get the call stack of a currently running thread and look up what task it ran at this time later on.

An easier and much more comfortable way to do that is the following x_cons command:

x_cons <DBSID> debugtask <Task-ID>

This command triggers the MaxDB Kernel to write out the current call stack for task <Task-ID> to the knldiag file.

So let us give it a try! I used my 7.5 Build 48 test database on windows and provoked a LOG FULL situation by using a statement like this:

create table test as select 'xxx', 'zzz', 0 from tables t1, tables t2

As soon as the database got in LOG FULL I used x_cons to figure out the task id of the hanging task:

x_cons db750 show active user   (ok I used the short form…)

C:\WINDOWS>x_cons db750 sh ta us
SERVERDB: DB750
ID   UKT  Win   TASK       APPL Current         Timeout Region     Wait
          tid   type        pid state          priority cnt try    item
T15    6  0xDEC User       3516 LOG FULL (246)        0 0               303(s)

So task 15 is the one we are looking for.
To get the call stack now, all I had to do is to issue this command:

x_cons db750 debugtask T15

That is it.

In the knldiag file, there is now the complete call stack as if it would be there when the database crashes. It looks like this (for readability, I omitted the timestamp entries):

[...]
0xF24     20001 Log      Log is full.
0xDEC WRN 20054 Log      LOG FULL: task 15 suspended
0xF24     20001 Log      Log is full.
0xDEC ERR 18891 CALLSTCK Dumping callstack of T15
0xF40 ERR 19999 BTRACE   Using 'imagehlp.dll' version: 4.0.5 
0xF40 ERR 19999 BTRACE   SymbolSearchPath:  c:\sapdb\data\wrk\DB750;C:\SAPDB\DB750\DB\pgm;C:\SAPDB\DB750\DB\symbols;C:\SAPDB\DB750\DB\sap;C:
0xF40 ERR 19999 BTRACE                      \WINDOWS;C:\SAPDB\DB750\DB\sap\
0xF40 ERR 19999 BTRACE   ----> Register Dump <----
0xF40 ERR 19999 BTRACE     Eax=0x009a0e20 Ebx=0x00000000 Ecx=0x02ae0d7c Edx=0x00000158 Esi=0x000001ac Edi=0x00000000
0xF40 ERR 19999 BTRACE     Eip=0x7c90eb94 Esp=0x02ae0d24 Ebp=0x02ae0d88
0xF40 ERR 19999 BTRACE     Cs=0x001b  Ss=0x0023  Ds=0x0023  Es=0x0023  Fs=0x003b  Gs=0x0000  Efl=0x0246
0xF40 ERR 19999 BTRACE   ----> Module List <----
0xF40 ERR 19999 BTRACE   |.text Start |.text End   | Module File Name
0xF40 ERR 19999 BTRACE   | 0x00400000 | 0x009d5000 | C:\\SAPDB\\DB750\\DB\\pgm\\kernel.exe
0xF40 ERR 19999 BTRACE   | 0x038b0000 | 0x03904000 | C:\\WINDOWS\\system32\\MSVCR70.dll
0xF40 ERR 19999 BTRACE   | 0x03920000 | 0x03a20000 | C:\\SAPDB\\DB750\\DB\\pgm\\dbghelp.dll
0xF40 ERR 19999 BTRACE   | 0x5d090000 | 0x5d12a000 | C:\\WINDOWS\\system32\\comctl32.dll
0xF40 ERR 19999 BTRACE   | 0x629c0000 | 0x629c9000 | C:\\WINDOWS\\system32\\LPK.DLL
0xF40 ERR 19999 BTRACE   | 0x71aa0000 | 0x71aa8000 | C:\\WINDOWS\\system32\\WS2HELP.dll
0xF40 ERR 19999 BTRACE   | 0x71ab0000 | 0x71ac7000 | C:\\WINDOWS\\system32\\WS2_32.dll
0xF40 ERR 19999 BTRACE   | 0x74d90000 | 0x74dfb000 | C:\\WINDOWS\\system32\\USP10.dll
0xF40 ERR 19999 BTRACE   | 0x76000000 | 0x7600d000 | C:\\sapdb\\programs\\pgm\\sqltcp1.dll
0xF40 ERR 19999 BTRACE   | 0x76050000 | 0x760c5000 | C:\\SAPDB\\DB750\\DB\\pgm\\liboms.dll
0xF40 ERR 19999 BTRACE   | 0x76390000 | 0x763ad000 | C:\\WINDOWS\\system32\\IMM32.DLL
0xF40 ERR 19999 BTRACE   | 0x76c90000 | 0x76cb8000 | C:\\WINDOWS\\system32\\imagehlp.dll
0xF40 ERR 19999 BTRACE   | 0x76f20000 | 0x76f47000 | C:\\WINDOWS\\system32\\DNSAPI.dll
0xF40 ERR 19999 BTRACE   | 0x76fc0000 | 0x76fc6000 | C:\\WINDOWS\\system32\\rasadhlp.dll
0xF40 ERR 19999 BTRACE   | 0x773d0000 | 0x774d3000 | C:\\WINDOWS\\WinSxS\\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2982_
0xF40 ERR 19999 BTRACE   :            :            : x-ww_ac3f9c03\\comctl32.dll
0xF40 ERR 19999 BTRACE   | 0x774e0000 | 0x7761d000 | C:\\WINDOWS\\system32\\ole32.dll
0xF40 ERR 19999 BTRACE   | 0x77c10000 | 0x77c68000 | C:\\WINDOWS\\system32\\msvcrt.dll
0xF40 ERR 19999 BTRACE   | 0x77dd0000 | 0x77e6b000 | C:\\WINDOWS\\system32\\ADVAPI32.dll
0xF40 ERR 19999 BTRACE   | 0x77e70000 | 0x77f02000 | C:\\WINDOWS\\system32\\RPCRT4.dll
0xF40 ERR 19999 BTRACE   | 0x77f10000 | 0x77f57000 | C:\\WINDOWS\\system32\\GDI32.dll
0xF40 ERR 19999 BTRACE   | 0x77f60000 | 0x77fd6000 | C:\\WINDOWS\\system32\\SHLWAPI.dll
0xF40 ERR 19999 BTRACE   | 0x77fe0000 | 0x77ff1000 | C:\\WINDOWS\\system32\\Secur32.dll
0xF40 ERR 19999 BTRACE   | 0x7c800000 | 0x7c8f5000 | C:\\WINDOWS\\system32\\kernel32.dll
0xF40 ERR 19999 BTRACE   | 0x7c900000 | 0x7c9b0000 | C:\\WINDOWS\\system32\\ntdll.dll
0xF40 ERR 19999 BTRACE   | 0x7c9c0000 | 0x7d1d6000 | C:\\WINDOWS\\system32\\SHELL32.dll
0xF40 ERR 19999 BTRACE   | 0x7e410000 | 0x7e4a0000 | C:\\WINDOWS\\system32\\USER32.dll
0xF40 ERR 19999 BTRACE   ----> Stack Back Trace <----
0xF40 ERR 19999 BTRACE     ntdll!KiFastSystemCallRet
0xF40 ERR 19999 BTRACE         SFrame:  IP:7c90eb94 RA:7c802532 FP:02ae0d88 SP:02ae0d24 
0xF40 ERR 19999 BTRACE         Params:  01:000001ac 02:ffffffff 03:00000000 04:02ae0dc8 
0xF40 ERR 19999 BTRACE     kernel32!WaitForSingleObject + 18 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:7c802532 RA:007b092d FP:02ae0d9c SP:02ae0d24 
0xF40 ERR 19999 BTRACE         Params:  01:000001ac 02:ffffffff 03:00000000 04:00000000 
0xF40 ERR 19999 BTRACE     kernel!eo670_WindowsTraceStack + 285 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007b092d RA:007b0af9 FP:02ae0dc8 SP:02ae0dd0 
0xF40 ERR 19999 BTRACE         Params:  01:00000000 02:00000000 03:7c800000 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  veo670-k.c ( Line: 7641 )
0xF40 ERR 19999 BTRACE     kernel!eo670_CTraceStackOCB + 73 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007b0af9 RA:007b0be2 FP:02ae0de4 SP:02ae0dec 
0xF40 ERR 19999 BTRACE         Params:  01:00000000 02:00000000 03:007d0258 04:00000005 
0xF40 ERR 19999 BTRACE         Source:  veo670-k.c ( Line: 1086 )
0xF40 ERR 19999 BTRACE     kernel!eo670_CTraceStack + 18 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007b0be2 RA:007d0258 FP:02ae0df0 SP:02ae0df8 
0xF40 ERR 19999 BTRACE         Params:  01:00000005 02:7ff15dc0 03:7c910340 04:02ae0e20 
0xF40 ERR 19999 BTRACE         Source:  veo670-k.c ( Line: 1048 )
0xF40 ERR 19999 BTRACE     kernel!sql74k_on_task_debug_request + 136 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007d0258 RA:007a61c9 FP:02ae0e04 SP:02ae0e0c 
0xF40 ERR 19999 BTRACE         Params:  01:02ae0e28 02:0000000f 03:00822a2c 04:ffffffff 
0xF40 ERR 19999 BTRACE         Source:  vos74k.c ( Line: 439 )
0xF40 ERR 19999 BTRACE     kernel!vsuspend + 329 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007a61c9 RA:0072897c FP:02ae0e20 SP:02ae0e28 
0xF40 ERR 19999 BTRACE         Params:  01:7fec0e88 02:000000f6 03:02b07a60 04:7ef8e778 
0xF40 ERR 19999 BTRACE         Source:  vos56k.c ( Line: 220 )
0xF40 ERR 19999 BTRACE     kernel!Log_Queue::WaitUntilSpaceIsAvailable + 828 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0072897c RA:006e35c8 FP:02ae0fdc SP:02ae0fe4 
0xF40 ERR 19999 BTRACE         Params:  01:0000000f 02:02b07a60 03:00000001 04:7f0b6934 
0xF40 ERR 19999 BTRACE         Source:  log_queue.cpp ( Line: 345 )
0xF40 ERR 19999 BTRACE     kernel!Log_Queue::Reserve + 40 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:006e35c8 RA:0071ad10 FP:02ae0ffc SP:02ae1004 
0xF40 ERR 19999 BTRACE         Params:  01:0000000f 02:02b07a60 03:0000005c 04:00000030 
0xF40 ERR 19999 BTRACE         Source:  log_queue.hpp ( Line: 195 )
0xF40 ERR 19999 BTRACE     kernel!Log_Transaction::WriteRedoEntry + 1472 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0071ad10 RA:00703ee0 FP:02ae1194 SP:02ae119c 
0xF40 ERR 19999 BTRACE         Params:  01:02ae11ac 02:7f0b6934 03:02b06c30 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  log_transaction.cpp ( Line: 1770 )
0xF40 ERR 19999 BTRACE     kernel!Log_ReadWriteActionImage::WriteAfterImage + 128 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00703ee0 RA:00668225 FP:02ae11e8 SP:02ae11f0 
0xF40 ERR 19999 BTRACE         Params:  01:02b079fc 02:02ae1214 03:0065e6f6 04:7f0b68b0 
0xF40 ERR 19999 BTRACE         Source:  log_readwriteactionimage.cpp ( Line: 221 )
0xF40 ERR 19999 BTRACE     kernel!kb611ins_WriteAfterImage + 21 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00668225 RA:0065e6f6 FP:02ae11f4 SP:02ae11fc 
0xF40 ERR 19999 BTRACE         Params:  01:7f0b68b0 02:02b079fc 03:02b06c30 04:02b06c30 
0xF40 ERR 19999 BTRACE         Source:  vkb611.cpp ( Line: 125 )
0xF40 ERR 19999 BTRACE     kernel!kb61insert_rec + 246 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0065e6f6 RA:0065f0cf FP:02ae1214 SP:02ae121c 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02ae1234 03:02aea1e8 04:00000002 
0xF40 ERR 19999 BTRACE         Source:  vkb61.c ( Line: 2639 )
0xF40 ERR 19999 BTRACE     kernel!k61ins_select + 303 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0065f0cf RA:00636235 FP:02ae1640 SP:02ae1648 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02aea1e8 03:00000001 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vkb61.c ( Line: 2147 )
0xF40 ERR 19999 BTRACE     kernel!kb721result_handle + 597 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00636235 RA:006375c6 FP:02ae1670 SP:02ae1678 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02ae21e8 03:02aed364 04:02ae1788 
0xF40 ERR 19999 BTRACE         Source:  vkb721.c ( Line: 3404 )
0xF40 ERR 19999 BTRACE     kernel!kb721select_gets + 342 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:006375c6 RA:00637836 FP:02ae1700 SP:02ae1708 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02aec700 03:02aed364 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vkb721.c ( Line: 2713 )
0xF40 ERR 19999 BTRACE     kernel!kb721seq_search + 614 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00637836 RA:006382d2 FP:02ae1738 SP:02ae1740 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02aed364 03:02ae21e8 04:02ae1788 
0xF40 ERR 19999 BTRACE         Source:  vkb721.c ( Line: 2806 )
0xF40 ERR 19999 BTRACE     kernel!k721strat_distribution + 370 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:006382d2 RA:00633f82 FP:02ae175c SP:02ae1764 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02aed364 03:02ae21e8 04:02ae1788 
0xF40 ERR 19999 BTRACE         Source:  vkb721.c ( Line: 4308 )
0xF40 ERR 19999 BTRACE     kernel!k720_select + 866 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00633f82 RA:00634154 FP:02ae21d4 SP:02ae21dc 
0xF40 ERR 19999 BTRACE         Params:  01:01b06c30 02:02aed364 03:02ae21e8 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vkb720.c ( Line: 2390 )
0xF40 ERR 19999 BTRACE     kernel!k720_single_select + 36 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00634154 RA:0065e074 FP:02af13a4 SP:02af13ac 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02b06c30 03:02af148c 04:00626e1a 
0xF40 ERR 19999 BTRACE         Source:  vkb720.c ( Line: 3461 )
0xF40 ERR 19999 BTRACE     kernel!k61table_ins_del_upd + 52 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0065e074 RA:00626e1a FP:02af13b4 SP:02af13bc 
0xF40 ERR 19999 BTRACE         Params:  01:02b06c30 02:02b079fc 03:02b06a44 04:02af19f0 
0xF40 ERR 19999 BTRACE         Source:  vkb61.c ( Line: 2303 )
0xF40 ERR 19999 BTRACE     kernel!k05functions + 1450 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00626e1a RA:0046c905 FP:02af148c SP:02af1494 
0xF40 ERR 19999 BTRACE         Params:  01:24b06c30 02:02b06c30 03:02b06a44 04:02af19f0 
0xF40 ERR 19999 BTRACE         Source:  vkb05.c ( Line: 2089 )
0xF40 ERR 19999 BTRACE     kernel!a06lsend_mess_buf + 821 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0046c905 RA:0046d49a FP:02af18c8 SP:02af18d0 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:24b06c30 03:00000000 04:02af19f0 
0xF40 ERR 19999 BTRACE         Source:  vak06.c ( Line: 7640 )
0xF40 ERR 19999 BTRACE     kernel!a06dml_send_mess_buf + 58 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0046d49a RA:00584a5b FP:02af18ec SP:02af18f4 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02b06c30 03:02af433c 04:02af19f0 
0xF40 ERR 19999 BTRACE         Source:  vak06.c ( Line: 7241 )
0xF40 ERR 19999 BTRACE     kernel!a505most_execute + 2731 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00584a5b RA:0058540b FP:02af19f8 SP:02af1a00 
0xF40 ERR 19999 BTRACE         Params:  01:00b06a44 02:02af433c 03:02af3318 04:02af27e0 
0xF40 ERR 19999 BTRACE         Source:  vak505.c ( Line: 6362 )
0xF40 ERR 19999 BTRACE     kernel!a505loop_most_execute + 811 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0058540b RA:00569f7b FP:02af1bd0 SP:02af1bd8 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02af433c 03:02af3318 04:02af269c 
0xF40 ERR 19999 BTRACE         Source:  vak505.c ( Line: 5892 )
0xF40 ERR 19999 BTRACE     kernel!a501exec_with_change_rec + 875 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00569f7b RA:00567e72 FP:02af26dc SP:02af26e4 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02af433c 03:02af27e0 04:02af3318 
0xF40 ERR 19999 BTRACE         Source:  vak501.c ( Line: 5344 )
0xF40 ERR 19999 BTRACE     kernel!a502complex_execution + 786 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00567e72 RA:00569e71 FP:02af27f8 SP:02af2800 
0xF40 ERR 19999 BTRACE         Params:  01:00b06a44 02:02af433c 03:02af3318 04:02b05eb4 
0xF40 ERR 19999 BTRACE         Source:  vak502.c ( Line: 5321 )
0xF40 ERR 19999 BTRACE     kernel!a501exec_with_change_rec + 609 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00569e71 RA:0056a252 FP:02af32f4 SP:02af32fc 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02af433c 03:02b05eb4 04:02af3318 
0xF40 ERR 19999 BTRACE         Source:  vak501.c ( Line: 5304 )
0xF40 ERR 19999 BTRACE     kernel!a501do_execute + 82 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0056a252 RA:0057b594 FP:02af431c SP:02af4324 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02af433c 03:02b05eb4 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vak501.c ( Line: 5395 )
0xF40 ERR 19999 BTRACE     kernel!a562_select_insert + 388 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0057b594 RA:004c80b9 FP:02b05ec8 SP:02b05ed0 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:00000000 03:02b06f88 04:02b06a44 
0xF40 ERR 19999 BTRACE         Source:  vak562.c ( Line: 7157 )
0xF40 ERR 19999 BTRACE     kernel!ak35call_semantik + 873 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:004c80b9 RA:004c8aca FP:02b05f18 SP:02b05f20 
0xF40 ERR 19999 BTRACE         Params:  01:00b06a44 02:02b072e2 03:00000000 04:02b06a44 
0xF40 ERR 19999 BTRACE         Source:  vak35.c ( Line: 5650 )
0xF40 ERR 19999 BTRACE     kernel!a35_asql_statement + 1674 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:004c8aca RA:00587fc6 FP:02b05f6c SP:02b05f74 
0xF40 ERR 19999 BTRACE         Params:  01:00b06a44 02:00000000 03:02b072e2 04:02b06a44 
0xF40 ERR 19999 BTRACE         Source:  vak35.c ( Line: 5247 )
0xF40 ERR 19999 BTRACE     kernel!ak92not_only_execute + 1574 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00587fc6 RA:00589ae1 FP:02b060d0 SP:02b060d8 
0xF40 ERR 19999 BTRACE         Params:  01:00b06a44 02:02b06220 03:02b0623a 04:02b061f8 
0xF40 ERR 19999 BTRACE         Source:  vak92.c ( Line: 5309 )
0xF40 ERR 19999 BTRACE     kernel!ak92analyze_messagetype + 1361 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00589ae1 RA:00589e17 FP:02b0622c SP:02b06234 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:00000000 03:00000000 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vak92.c ( Line: 5727 )
0xF40 ERR 19999 BTRACE     kernel!a92_mode_analyzer + 279 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00589e17 RA:0047c034 FP:02b0630c SP:02b06314 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:00000000 03:00000000 04:02b06a44 
0xF40 ERR 19999 BTRACE         Source:  vak92.c ( Line: 6401 )
0xF40 ERR 19999 BTRACE     kernel!ak14insert_select + 228 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0047c034 RA:0047d5b0 FP:02b06348 SP:02b06350 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02b06458 03:00000002 04:0000002b 
0xF40 ERR 19999 BTRACE         Source:  vak14.c ( Line: 5307 )
0xF40 ERR 19999 BTRACE     kernel!a14create_tab_as_select + 592 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0047d5b0 RA:0058f39f FP:02b0647c SP:02b06484 
0xF40 ERR 19999 BTRACE         Params:  01:02b06a44 02:02b06568 03:00020000 04:00000011 
0xF40 ERR 19999 BTRACE         Source:  vak14.c ( Line: 5038 )
0xF40 ERR 19999 BTRACE     kernel!ak93one_command + 1567 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:0058f39f RA:00590516 FP:02b0658c SP:02b06594 
0xF40 ERR 19999 BTRACE         Params:  01:01b06a44 02:00000001 03:02b065f0 04:00000001 
0xF40 ERR 19999 BTRACE         Source:  vak93.c ( Line: 6525 )
0xF40 ERR 19999 BTRACE     kernel!a93_user_commands + 1046 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00590516 RA:004487d6 FP:02b06604 SP:02b0660c 
0xF40 ERR 19999 BTRACE         Params:  01:00b06a44 02:02b06a3b 03:02b06a2f 04:02b07f6c 
0xF40 ERR 19999 BTRACE         Source:  vak93.c ( Line: 5603 )
0xF40 ERR 19999 BTRACE     kernel!ak91run_sql_process + 134 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:004487d6 RA:004488f2 FP:02b06a30 SP:02b06a38 
0xF40 ERR 19999 BTRACE         Params:  01:01b06a44 02:00000001 03:00000001 04:7ef9252c 
0xF40 ERR 19999 BTRACE         Source:  vak91.c ( Line: 5406 )
0xF40 ERR 19999 BTRACE     kernel!ak91run_user_process + 98 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:004488f2 RA:00448989 FP:02b07ee4 SP:02b07eec 
0xF40 ERR 19999 BTRACE         Params:  01:02b07f6c 02:00000003 03:7ff70818 04:02b07f54 
0xF40 ERR 19999 BTRACE         Source:  vak91.c ( Line: 5434 )
0xF40 ERR 19999 BTRACE     kernel!a91mainprogam_with_allocator + 57 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00448989 RA:006d675f FP:02b07ef8 SP:02b07f00 
0xF40 ERR 19999 BTRACE         Params:  01:03b07f6c 02:7c910340 03:7fec0e88 04:7c910331 
0xF40 ERR 19999 BTRACE         Source:  vak91.c ( Line: 5459 )
0xF40 ERR 19999 BTRACE     kernel!gg941CreateAllocatorAndCallMainprog + 223 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:006d675f RA:00448923 FP:02b07f54 SP:02b07f5c 
0xF40 ERR 19999 BTRACE         Params:  01:02b07f03 02:02b07fb4 03:00000014 04:02b07f6c 
0xF40 ERR 19999 BTRACE         Source:  vgg941.cpp ( Line: 156 )
0xF40 ERR 19999 BTRACE     kernel!a91mainprogram + 35 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:00448923 RA:007d3054 FP:02b07fe0 SP:02b07fe8 
0xF40 ERR 19999 BTRACE         Params:  01:7fec0e88 02:007d3854 03:00000000 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vak91.c ( Line: 5446 )
0xF40 ERR 19999 BTRACE     kernel!sql88k_kernel_task + 148 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007d3054 RA:007d3854 FP:02b07fe8 SP:02b07ff0 
0xF40 ERR 19999 BTRACE         Params:  01:00000000 02:00000000 03:00000000 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vos88k.c ( Line: 416 )
0xF40 ERR 19999 BTRACE     kernel!sql88k_ukt + 1140 bytes
0xF40 ERR 19999 BTRACE         SFrame:  IP:007d3854 RA:00000000 FP:02b07ff0 SP:02b07ff0 
0xF40 ERR 19999 BTRACE         Params:  01:00000000 02:00000000 03:00000000 04:00000000 
0xF40 ERR 19999 BTRACE         Source:  vos88k.c ( Line: 277 )
------------------- current write position ----------------

As you can see, it is pretty much the same content as you get from the windows debugger.
The call stack is again written top-down, putting the last executed calls in the first rows of the output. As we see the call stack routine was the last ‘thing’ the task ran through – not too surprising.
Anyhow, when we browse a bit further below, we find
kernel!Log_Queue::WaitUntilSpaceIsAvailable
so we can tell what the task is currently doing when it is not writing call stacks.
For a more elaborate discussion of a call stack visit my first blog entry on this topic (see link above).

By the way: very often in customer messages people (people and collegues as well) try to give valueable information by attaching parts of this call stack to the message. Unfortunately most people seem to believe, that the “—-> Module List <—-“ and “—-> Register Dump <—-“ are the most critical information and forget to attach the “—-> Stack Back Trace <—-“ as well.
In fact it’s the exact opposite.While the stack back trace is somwhat intelligible to supporters, the module list and the register dump contain information that only a few developers can make sense of.

As far as I have tested, this call works for MaxDBs from version 7.4.3 onwards.
As the debugtask command is not documented, it is an unsupported function and should not be used on productive systems unless advised by the SAP support.
Nevertheless, it may be sometime interesting to look at the call stack for educational purposes.

Best regards,
Lars