Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4993 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.9336. Min Act: -0.0000

Data Index: 1542040 (Python Code)

Max Activating Token Index: 791

Text #1

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.4758. Min Act: -0.0000

Data Index: 1587156 (Python Code)

Max Activating Token Index: 518

Text #2

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.4736. Min Act: -0.0000

Data Index: 1630188 (Python Code)

Max Activating Token Index: 779

Text #3

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.1935. Min Act: -0.0000

Data Index: 1121560 (C4 (Web Text))

Max Activating Token Index: 996

Text #4

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.3208. Min Act: -0.0000

Data Index: 1618335 (Python Code)

Max Activating Token Index: 492

Text #5

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.4801. Min Act: -0.0000

Data Index: 1465081 (Python Code)

Max Activating Token Index: 809

Text #6

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.4371. Min Act: -0.0000

Data Index: 1470386 (Python Code)

Max Activating Token Index: 956

Text #7

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.2328. Min Act: -0.0000

Data Index: 1540158 (Python Code)

Max Activating Token Index: 921

Text #8

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.0116. Min Act: -0.0000

Data Index: 1528657 (Python Code)

Max Activating Token Index: 532

Text #9

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.3806. Min Act: -0.0000

Data Index: 1370156 (Python Code)

Max Activating Token Index: 783

Text #10

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.1457. Min Act: -0.0000

Data Index: 1518372 (Python Code)

Max Activating Token Index: 944

Text #11

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.9989. Min Act: -0.0000

Data Index: 1304471 (C4 (Web Text))

Max Activating Token Index: 861

Text #12

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.0612. Min Act: -0.0000

Data Index: 483803 (C4 (Web Text))

Max Activating Token Index: 959

Text #13

Max Range: 6.9336. Min Range: -6.9336

Max Act: 6.2138. Min Act: -0.0000

Data Index: 1576231 (Python Code)

Max Activating Token Index: 686

Text #14

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.8016. Min Act: -0.0000

Data Index: 1425178 (Python Code)

Max Activating Token Index: 598

Text #15

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.9244. Min Act: -0.0000

Data Index: 1541722 (Python Code)

Max Activating Token Index: 462

Text #16

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.7139. Min Act: -0.0000

Data Index: 1621702 (Python Code)

Max Activating Token Index: 476

Text #17

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.9221. Min Act: -0.0000

Data Index: 1435315 (Python Code)

Max Activating Token Index: 745

Text #18

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.6691. Min Act: -0.0000

Data Index: 1403006 (Python Code)

Max Activating Token Index: 954

Text #19

Max Range: 6.9336. Min Range: -6.9336

Max Act: 5.8208. Min Act: -0.0000

Data Index: 1408602 (Python Code)

Max Activating Token Index: 820