Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4588 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2789. Min Range: -2.2789

Max Act: 2.1714. Min Act: -0.0001

Data Index: 90366 (C4 (Web Text))

Max Activating Token Index: 89

Text #1

Max Range: 2.2789. Min Range: -2.2789

Max Act: 2.2789. Min Act: -0.0001

Data Index: 874615 (C4 (Web Text))

Max Activating Token Index: 61

Text #2

Max Range: 2.2789. Min Range: -2.2789

Max Act: 2.0168. Min Act: -0.0000

Data Index: 316774 (C4 (Web Text))

Max Activating Token Index: 37

Text #3

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.8880. Min Act: -0.0001

Data Index: 1134949 (C4 (Web Text))

Max Activating Token Index: 259

Text #4

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.6906. Min Act: -0.0001

Data Index: 105619 (C4 (Web Text))

Max Activating Token Index: 216

Text #5

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.8422. Min Act: -0.0001

Data Index: 399191 (C4 (Web Text))

Max Activating Token Index: 936

Text #6

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.7807. Min Act: -0.0001

Data Index: 310259 (C4 (Web Text))

Max Activating Token Index: 772

Text #7

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.6094. Min Act: -0.0000

Data Index: 923113 (C4 (Web Text))

Max Activating Token Index: 390

Text #8

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4064. Min Act: -0.0001

Data Index: 765281 (C4 (Web Text))

Max Activating Token Index: 441

Text #9

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.6663. Min Act: -0.0000

Data Index: 318917 (C4 (Web Text))

Max Activating Token Index: 478

Text #10

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.5671. Min Act: -0.0001

Data Index: 1298596 (C4 (Web Text))

Max Activating Token Index: 98

Text #11

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4355. Min Act: -0.0000

Data Index: 1275501 (C4 (Web Text))

Max Activating Token Index: 447

Text #12

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4950. Min Act: -0.0000

Data Index: 1433255 (Python Code)

Max Activating Token Index: 5

Text #13

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.1370. Min Act: -0.0000

Data Index: 1113630 (C4 (Web Text))

Max Activating Token Index: 140

Text #14

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4607. Min Act: -0.0000

Data Index: 1633238 (Python Code)

Max Activating Token Index: 5

Text #15

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4589. Min Act: -0.0000

Data Index: 998370 (C4 (Web Text))

Max Activating Token Index: 885

Text #16

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4389. Min Act: -0.0001

Data Index: 1464855 (Python Code)

Max Activating Token Index: 6

Text #17

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.5084. Min Act: -0.0001

Data Index: 851253 (C4 (Web Text))

Max Activating Token Index: 222

Text #18

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.4248. Min Act: -0.0000

Data Index: 898450 (C4 (Web Text))

Max Activating Token Index: 443

Text #19

Max Range: 2.2789. Min Range: -2.2789

Max Act: 1.5574. Min Act: -0.0001

Data Index: 344543 (C4 (Web Text))

Max Activating Token Index: 287