This is a follow-up to a problem I had with yesterday’s example of using recursive CTEs to “bounce” along a multi-column index to pick out the unique set of combinations of the first two columns. Part of the resulting query used a pair of aggregate scalar subqueries in a select list – and Andrew Sayer improved on my query by introducing a “cross apply” (which I simply hadn’t thought of) which the optimizer transformed into a lateral view (which I had thought of, but couldn’t get to work).
After seeing what the Andrew and the optimizer had done I looked a little more closely at my lateral view experiment and modified it so that it worked. Here are the three critical versions of the relevant code fragment; first is my original code, then Andrew’s cross apply, then my working lateral view version:
select
(select min(t1.val1) val1 from t1 where t1.val1 > bounce1.val1) val1,
(select min(t1.val2) val2 from t1 where t1.val1 > bounce1.val1 and rownum = 1) val2
from bounce1
where bounce1.val1 is not null
select
ca.val1 ,ca.val2
from bounce1
cross apply (select val1, val2
from (select /*+ index(t1) no_index_ffs(t1) */
val1, val2
from t1
where t1.val1 > bounce1.val1
and rownum = 1
)
) ca
where bounce1.val1 is not null
----
select
ca.val1 ,ca.val2
from bounce1,
lateral(select val1, val2
from (select /*+ index(t1) no_index_ffs(t1) */
val1, val2
from t1
where t1.val1 > bounce1.val1
and rownum = 1
)
) ca
where bounce1.val1 is not null
All I’ve done to modify Andrew’s code is put a comma after the table (actually CTE) bounce1, then change “cross apply” to “lateral”. Compare the resulting text with the following lateral version that doesn’t work:
select
ca.val1 ,ca.val2
from bounce1,
lateral (
select /*+ index(t1) no_index_ffs(t1) */
val1, val2
from t1
where t1.val1 > bounce1.val1
and rownum = 1
) ca
where bounce1.val1 is not null
To get from not working to working all I’ve done is wrap the text in my lateral() subquery inside one more (apparently redundant) layer of “select * from ()”!
In fact my assumption that my code wasn’t working was incorrect – what was really going on was that the code I had written was producing the wrong results but I thought that I had made a mistake in the way I was writing it and couldn’t figure out what I had done wrong.
Problem Solving:
To get a better idea of what’s going on, I took a closer look at the execution plans. Here are the plans (main body only) for the two variants of using the lateral() view – the first from the SQL with the “redundant” select, the second as I originally wrote it. Notice that the number of rows (A-Rows) returned in the first case is the 30 expected while in the second case it’s only 10.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Starts | E-Rows | Cost (%CPU)| A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | | 125 (100)| 30 |00:00:00.01 | 40 | 28 | | | | | 1 | SORT ORDER BY | | 1 | 4 | 125 (3)| 30 |00:00:00.01 | 40 | 28 | 2048 | 2048 | 2048 (0)| |* 2 | VIEW | | 1 | 4 | 124 (2)| 30 |00:00:00.01 | 40 | 28 | | | | | 3 | UNION ALL (RECURSIVE WITH) BREADTH FIRST | | 1 | | | 33 |00:00:00.01 | 40 | 28 | 1024 | 1024 | | |* 4 | VIEW | | 1 | 2 | 61 (2)| 3 |00:00:00.01 | 8 | 4 | | | | | 5 | UNION ALL (RECURSIVE WITH) BREADTH FIRST| | 1 | | | 3 |00:00:00.01 | 8 | 4 | 1024 | 1024 | | |* 6 | VIEW | | 1 | 1 | 29 (0)| 1 |00:00:00.01 | 2 | 1 | | | | |* 7 | WINDOW NOSORT STOPKEY | | 1 | 10000 | 29 (0)| 1 |00:00:00.01 | 2 | 1 | 73728 | 73728 | | | 8 | INDEX FULL SCAN | T1_PK | 1 | 10000 | 29 (0)| 2 |00:00:00.01 | 2 | 1 | | | | | 9 | NESTED LOOPS | | 3 | 1 | 31 (0)| 2 |00:00:00.01 | 6 | 3 | | | | | 10 | RECURSIVE WITH PUMP | | 3 | | | 3 |00:00:00.01 | 0 | 0 | | | | | 11 | VIEW | VW_LAT_1BBF5C63 | 3 | 1 | 2 (0)| 2 |00:00:00.01 | 6 | 3 | | | | | 12 | VIEW | | 3 | 1 | 2 (0)| 2 |00:00:00.01 | 6 | 3 | | | | |* 13 | COUNT STOPKEY | | 3 | | | 2 |00:00:00.01 | 6 | 3 | | | | |* 14 | INDEX RANGE SCAN | T1_PK | 3 | 1 | 2 (0)| 2 |00:00:00.01 | 6 | 3 | | | | | 15 | SORT AGGREGATE | | 30 | 1 | | 30 |00:00:00.01 | 32 | 24 | | | | | 16 | FIRST ROW | | 30 | 1 | 2 (0)| 27 |00:00:00.01 | 32 | 24 | | | | |* 17 | INDEX RANGE SCAN (MIN/MAX) | T1_PK | 30 | 1 | 2 (0)| 27 |00:00:00.01 | 32 | 24 | | | | | 18 | RECURSIVE WITH PUMP | | 11 | | | 30 |00:00:00.01 | 0 | 0 | | | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Id | Operation | Name | Starts | E-Rows | Cost (%CPU)| A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | | 125 (100)| 10 |00:00:00.01 | 16 | | | | | 1 | SORT ORDER BY | | 1 | 4 | 125 (3)| 10 |00:00:00.01 | 16 | 2048 | 2048 | 2048 (0)| |* 2 | VIEW | | 1 | 4 | 124 (2)| 10 |00:00:00.01 | 16 | | | | | 3 | UNION ALL (RECURSIVE WITH) BREADTH FIRST | | 1 | | | 11 |00:00:00.01 | 16 | 1024 | 1024 | | |* 4 | VIEW | | 1 | 2 | 61 (2)| 1 |00:00:00.01 | 4 | | | | | 5 | UNION ALL (RECURSIVE WITH) BREADTH FIRST| | 1 | | | 1 |00:00:00.01 | 4 | 1024 | 1024 | | |* 6 | VIEW | | 1 | 1 | 29 (0)| 1 |00:00:00.01 | 2 | | | | |* 7 | WINDOW NOSORT STOPKEY | | 1 | 10000 | 29 (0)| 1 |00:00:00.01 | 2 | 73728 | 73728 | | | 8 | INDEX FULL SCAN | T1_PK | 1 | 10000 | 29 (0)| 2 |00:00:00.01 | 2 | | | | | 9 | NESTED LOOPS | | 1 | 1 | 31 (0)| 0 |00:00:00.01 | 2 | | | | | 10 | RECURSIVE WITH PUMP | | 1 | | | 1 |00:00:00.01 | 0 | | | | |* 11 | VIEW | VW_DCL_1BBF5C63 | 1 | 1 | 2 (0)| 0 |00:00:00.01 | 2 | | | | |* 12 | COUNT STOPKEY | | 1 | | | 1 |00:00:00.01 | 2 | | | | | 13 | INDEX FULL SCAN | T1_PK | 1 | 1 | 2 (0)| 1 |00:00:00.01 | 2 | | | | | 14 | SORT AGGREGATE | | 10 | 1 | | 10 |00:00:00.01 | 12 | | | | | 15 | FIRST ROW | | 10 | 1 | 2 (0)| 9 |00:00:00.01 | 12 | | | | |* 16 | INDEX RANGE SCAN (MIN/MAX) | T1_PK | 10 | 1 | 2 (0)| 9 |00:00:00.01 | 12 | | | | | 17 | RECURSIVE WITH PUMP | | 11 | | | 10 |00:00:00.01 | 0 | | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------
Most importantly we can see that the optimizer has used two different transformations. For the working query we see the view name VW_LAT_xxxxxxxx at operation 11, this is Oracle implementing a lateral view; for the problem query we see the view name VW_DCL_xxxxxxxx at operation 11, which is Oracle implementing a transformation to a “decorrelated lateral view”.
My first test after noting this difference was to see what would happen in I added the hint /*+ no_query_transformation */ to the query: it resulted in the VW_DCL_xxxxxxxx view name changing to VW_LAT_xxxxxxxx and the query producing the right result. Andrew Sayer, on the ODC thread, then pointed out that he’d done a couple more experiments and used the /*+ no_decorrelate() */ hint so I tried that with my query, adding it (with no parameters) to the subquery inside the lateral() clause – again the plan changed from using VW_DCL to VW_LAT and the results were correct.
Test Case
Bottom line on this – it looks like the optimizer is decorrelating a subquery when it shouldn’t, leading to wrong results. To make it easier to see this anomaly I stripped the original sample down to a basic test case starting with the table that I used in the previous posting:
rem
rem Script: decorralate.sql
rem Author: Jonathan Lewis
rem Dated: May 2018
rem
rem Last tested
rem 18.1.0.0 -- via liveSQL
rem 12.2.0.1
rem 12.1.0.2
rem
create table t1
segment creation immediate
nologging
as
with generator as (
select
rownum id
from dual
connect by
level <= 1e4 -- > comment to avoid WordPress format issue
)
select
rownum id,
mod(rownum-1,3) val1,
mod(rownum-1,10) val2,
lpad('x',100,'x') padding
from
generator v1
order by
dbms_random.value
;
begin
dbms_stats.gather_table_stats(
ownname => user,
tabname => 'T1',
method_opt => 'for all columns size 1'
);
end;
/
alter table t1 add constraint t1_pk primary key(val1, val2, id);
Now two versions of a simplified piece of code that should select the distinct values of val1 greater than the lowest value (each row in the UNION ALL of dual is emulating the way in which yesterday’s recursive CTE was effectively saying “this is a current known value, find the next higher”):
prompt =============
prompt Right results
prompt =============
select
v1.val1, v1.val2
from (
select 0 val1, 0 val2 from dual
union all
select 1,0 from dual
union all
select 2,0 from dual
) bounce1,
lateral (
select val1, val2 from (
select /*+ index(t1) no_index_ffs(t1) */
t1.val1, t1.val2
from t1
where t1.val1 > bounce1.val1
and rownum = 1
)
) v1
;
prompt ===========================================
prompt Wrong results -- "redundant" select removed
prompt ===========================================
select
v1.val1, v1.val2
from (
select 0 val1, 0 val2 from dual
union all
select 1,0 from dual
union all
select 2,0 from dual
) bounce1,
lateral (
-- select val1, val2 from (
select /*+ index(t1) no_index_ffs(t1) */
t1.val1, t1.val2
from t1
where t1.val1 > bounce1.val1
and rownum = 1
-- )
) v1
;
Here’s a cut-n-paste from running the two queries:
=============
Right results
=============
VAL1 VAL2
---------- ----------
1 0
2 0
2 rows selected.
============================================
Wrong results -- "redundant" select removed
============================================
no rows selected
Finally, to get an idea of what’s gone wrong – and to show that the optimizer has done something wrong when attempting to decorrelate – we can take a look at the optimizer trace file to see the final transformed SQL that the optimizer has produced a plan for. (I enabled the trace with the command “alter session set events ‘trace [rdbms.SQL_Transform.*]’;” to limit the trace to just the information about optimizer transformations.) This – cosmetically altered – is the final “unparsed” query:
select
vw_dcl_a18161ff.val1 val1,
vw_dcl_a18161ff.val2 val2
from (
(select 0 val1 from sys.dual dual)
union all
(select 1 1 from sys.dual dual)
union all
(select 2 2 from sys.dual dual)
) bounce1,
(
select
/*+ no_index_ffs (t1) index (t1) */
t1.val1 val1_0,
t1.val2 val2_1
from
test_user.t1 t1
where
rownum = 1
) vw_dcl_a18161ff
where
vw_dcl_a18161ff.val1 > bounce1.val1
As you can see, the lateral view has turned into a non-mergeable inline view which selects the first row available from t1 by following the supplied hints, and joins that single row result set to bounce1. I have a suspicion that lateral views which include rownum predicates should not be decorrelated. I have looked on MoS to see if I can find any bugs related to decorrelating lateral views, but either there are none or my search terms weren’t good enough.