Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3859 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.3247. Min Act: -0.0000

Data Index: 1416626 (Python Code)

Max Activating Token Index: 888

Text #1

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.4783. Min Act: -0.0000

Data Index: 1650761 (Python Code)

Max Activating Token Index: 948

Text #2

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.0770. Min Act: -0.0000

Data Index: 1554787 (Python Code)

Max Activating Token Index: 705

Text #3

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.3466. Min Act: -0.0000

Data Index: 1438131 (Python Code)

Max Activating Token Index: 43

Text #4

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.0079. Min Act: -0.0000

Data Index: 1419277 (Python Code)

Max Activating Token Index: 974

Text #5

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.2623. Min Act: -0.0000

Data Index: 1583183 (Python Code)

Max Activating Token Index: 296

Text #6

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.8944. Min Act: -0.0000

Data Index: 1490910 (Python Code)

Max Activating Token Index: 300

Text #7

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.0338. Min Act: -0.0000

Data Index: 1597572 (Python Code)

Max Activating Token Index: 277

Text #8

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.0043. Min Act: -0.0000

Data Index: 1435642 (Python Code)

Max Activating Token Index: 1016

Text #9

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.9812. Min Act: -0.0000

Data Index: 1632051 (Python Code)

Max Activating Token Index: 87

Text #10

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.9879. Min Act: -0.0000

Data Index: 1536647 (Python Code)

Max Activating Token Index: 157

Text #11

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.8249. Min Act: -0.0000

Data Index: 1654403 (Python Code)

Max Activating Token Index: 953

Text #12

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.7932. Min Act: -0.0000

Data Index: 1439803 (Python Code)

Max Activating Token Index: 97

Text #13

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.9319. Min Act: -0.0000

Data Index: 1386020 (Python Code)

Max Activating Token Index: 585

Text #14

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.7572. Min Act: -0.0000

Data Index: 1610283 (Python Code)

Max Activating Token Index: 498

Text #15

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.7567. Min Act: -0.0000

Data Index: 1637048 (Python Code)

Max Activating Token Index: 488

Text #16

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.0174. Min Act: -0.0000

Data Index: 1517975 (Python Code)

Max Activating Token Index: 893

Text #17

Max Range: 3.4783. Min Range: -3.4783

Max Act: 3.2084. Min Act: -0.0000

Data Index: 1468913 (Python Code)

Max Activating Token Index: 593

Text #18

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.7570. Min Act: -0.0000

Data Index: 1443973 (Python Code)

Max Activating Token Index: 351

Text #19

Max Range: 3.4783. Min Range: -3.4783

Max Act: 2.5979. Min Act: -0.0000

Data Index: 1424102 (Python Code)

Max Activating Token Index: 247