Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2167 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.3007. Min Act: -0.0003

Data Index: 1638702 (Python Code)

Max Activating Token Index: 581

Text #1

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2941. Min Act: -0.0003

Data Index: 1367100 (Python Code)

Max Activating Token Index: 501

Text #2

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2984. Min Act: -0.0003

Data Index: 22038 (C4 (Web Text))

Max Activating Token Index: 638

Text #3

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2763. Min Act: -0.0003

Data Index: 1590559 (Python Code)

Max Activating Token Index: 363

Text #4

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2852. Min Act: -0.0003

Data Index: 1596299 (Python Code)

Max Activating Token Index: 159

Text #5

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2760. Min Act: -0.0003

Data Index: 35967 (C4 (Web Text))

Max Activating Token Index: 203

Text #6

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2796. Min Act: -0.0003

Data Index: 1620908 (Python Code)

Max Activating Token Index: 315

Text #7

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2472. Min Act: -0.0003

Data Index: 1491381 (Python Code)

Max Activating Token Index: 301

Text #8

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2756. Min Act: -0.0003

Data Index: 1403233 (Python Code)

Max Activating Token Index: 528

Text #9

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2774. Min Act: -0.0003

Data Index: 1424216 (Python Code)

Max Activating Token Index: 198

Text #10

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2674. Min Act: -0.0003

Data Index: 1408316 (Python Code)

Max Activating Token Index: 231

Text #11

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2667. Min Act: -0.0003

Data Index: 1611679 (Python Code)

Max Activating Token Index: 366

Text #12

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2866. Min Act: -0.0003

Data Index: 1623862 (Python Code)

Max Activating Token Index: 206

Text #13

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2852. Min Act: -0.0003

Data Index: 1416770 (Python Code)

Max Activating Token Index: 314

Text #14

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2638. Min Act: -0.0003

Data Index: 1384157 (Python Code)

Max Activating Token Index: 323

Text #15

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2639. Min Act: -0.0003

Data Index: 1574108 (Python Code)

Max Activating Token Index: 148

Text #16

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2647. Min Act: -0.0003

Data Index: 1458124 (Python Code)

Max Activating Token Index: 235

Text #17

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2803. Min Act: -0.0003

Data Index: 1433208 (Python Code)

Max Activating Token Index: 171

Text #18

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2697. Min Act: -0.0003

Data Index: 1502268 (Python Code)

Max Activating Token Index: 546

Text #19

Max Range: 0.3007. Min Range: -0.3007

Max Act: 0.2675. Min Act: -0.0003

Data Index: 1486358 (Python Code)

Max Activating Token Index: 217