Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3338 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1912. Min Range: -2.1912

Max Act: 2.1912. Min Act: -0.0000

Data Index: 316723 (C4 (Web Text))

Max Activating Token Index: 12

Text #1

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.6832. Min Act: -0.0000

Data Index: 1439918 (Python Code)

Max Activating Token Index: 423

Text #2

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.5850. Min Act: -0.0000

Data Index: 1560559 (Python Code)

Max Activating Token Index: 590

Text #3

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.3542. Min Act: -0.0000

Data Index: 1607356 (Python Code)

Max Activating Token Index: 591

Text #4

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.4168. Min Act: -0.0000

Data Index: 1304959 (C4 (Web Text))

Max Activating Token Index: 116

Text #5

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.3519. Min Act: -0.0000

Data Index: 1574503 (Python Code)

Max Activating Token Index: 738

Text #6

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.2578. Min Act: -0.0000

Data Index: 1320793 (C4 (Web Text))

Max Activating Token Index: 74

Text #7

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.2954. Min Act: -0.0000

Data Index: 1453286 (Python Code)

Max Activating Token Index: 625

Text #8

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1195. Min Act: -0.0000

Data Index: 1461996 (Python Code)

Max Activating Token Index: 118

Text #9

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.2463. Min Act: -0.0000

Data Index: 602092 (C4 (Web Text))

Max Activating Token Index: 728

Text #10

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1684. Min Act: -0.0000

Data Index: 1014105 (C4 (Web Text))

Max Activating Token Index: 9

Text #11

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1285. Min Act: -0.0000

Data Index: 735693 (C4 (Web Text))

Max Activating Token Index: 536

Text #12

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1874. Min Act: -0.0000

Data Index: 1370366 (Python Code)

Max Activating Token Index: 990

Text #13

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1058. Min Act: -0.0000

Data Index: 1496377 (Python Code)

Max Activating Token Index: 12

Text #14

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.0781. Min Act: -0.0000

Data Index: 1404949 (Python Code)

Max Activating Token Index: 202

Text #15

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1730. Min Act: -0.0000

Data Index: 1389296 (Python Code)

Max Activating Token Index: 237

Text #16

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.2215. Min Act: -0.0000

Data Index: 1389999 (Python Code)

Max Activating Token Index: 298

Text #17

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.0440. Min Act: -0.0000

Data Index: 858649 (C4 (Web Text))

Max Activating Token Index: 199

Text #18

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.1257. Min Act: -0.0000

Data Index: 1173998 (C4 (Web Text))

Max Activating Token Index: 385

Text #19

Max Range: 2.1912. Min Range: -2.1912

Max Act: 1.0552. Min Act: -0.0000

Data Index: 1623976 (Python Code)

Max Activating Token Index: 849