Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3060 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4756. Min Range: -3.4756

Max Act: 3.4584. Min Act: -0.0000

Data Index: 1565720 (Python Code)

Max Activating Token Index: 387

Text #1

Max Range: 3.4756. Min Range: -3.4756

Max Act: 3.4206. Min Act: -0.0000

Data Index: 1637811 (Python Code)

Max Activating Token Index: 952

Text #2

Max Range: 3.4756. Min Range: -3.4756

Max Act: 3.4756. Min Act: -0.0000

Data Index: 1458352 (Python Code)

Max Activating Token Index: 896

Text #3

Max Range: 3.4756. Min Range: -3.4756

Max Act: 3.4308. Min Act: -0.0000

Data Index: 1487773 (Python Code)

Max Activating Token Index: 759

Text #4

Max Range: 3.4756. Min Range: -3.4756

Max Act: 3.3929. Min Act: -0.0000

Data Index: 1454241 (Python Code)

Max Activating Token Index: 396

Text #5

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.9382. Min Act: -0.0000

Data Index: 1374385 (Python Code)

Max Activating Token Index: 266

Text #6

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.8123. Min Act: -0.0000

Data Index: 1543376 (Python Code)

Max Activating Token Index: 970

Text #7

Max Range: 3.4756. Min Range: -3.4756

Max Act: 3.0359. Min Act: -0.0000

Data Index: 1502796 (Python Code)

Max Activating Token Index: 382

Text #8

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.8849. Min Act: -0.0000

Data Index: 1510448 (Python Code)

Max Activating Token Index: 874

Text #9

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.7700. Min Act: -0.0000

Data Index: 1142092 (C4 (Web Text))

Max Activating Token Index: 1010

Text #10

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.8824. Min Act: -0.0000

Data Index: 573549 (C4 (Web Text))

Max Activating Token Index: 83

Text #11

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.6714. Min Act: -0.0000

Data Index: 599245 (C4 (Web Text))

Max Activating Token Index: 962

Text #12

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.7848. Min Act: -0.0000

Data Index: 1016704 (C4 (Web Text))

Max Activating Token Index: 915

Text #13

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.5720. Min Act: -0.0000

Data Index: 1480931 (Python Code)

Max Activating Token Index: 967

Text #14

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.4127. Min Act: -0.0000

Data Index: 1477768 (Python Code)

Max Activating Token Index: 917

Text #15

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.7494. Min Act: -0.0000

Data Index: 1012388 (C4 (Web Text))

Max Activating Token Index: 854

Text #16

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.3425. Min Act: -0.0000

Data Index: 1381868 (Python Code)

Max Activating Token Index: 160

Text #17

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.3844. Min Act: -0.0000

Data Index: 1387363 (Python Code)

Max Activating Token Index: 191

Text #18

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.3594. Min Act: -0.0000

Data Index: 1632596 (Python Code)

Max Activating Token Index: 741

Text #19

Max Range: 3.4756. Min Range: -3.4756

Max Act: 2.3125. Min Act: -0.0000

Data Index: 1525536 (Python Code)

Max Activating Token Index: 481