Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5001 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8980. Min Range: -2.8980

Max Act: 2.8980. Min Act: -0.0000

Data Index: 325863 (C4 (Web Text))

Max Activating Token Index: 325

Text #1

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.9284. Min Act: -0.0000

Data Index: 744263 (C4 (Web Text))

Max Activating Token Index: 933

Text #2

Max Range: 2.8980. Min Range: -2.8980

Max Act: 2.5723. Min Act: -0.0000

Data Index: 1034253 (C4 (Web Text))

Max Activating Token Index: 925

Text #3

Max Range: 2.8980. Min Range: -2.8980

Max Act: 2.2300. Min Act: -0.0000

Data Index: 272287 (C4 (Web Text))

Max Activating Token Index: 414

Text #4

Max Range: 2.8980. Min Range: -2.8980

Max Act: 2.0354. Min Act: -0.0000

Data Index: 769723 (C4 (Web Text))

Max Activating Token Index: 376

Text #5

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.9354. Min Act: -0.0000

Data Index: 232396 (C4 (Web Text))

Max Activating Token Index: 375

Text #6

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.6200. Min Act: -0.0000

Data Index: 131829 (C4 (Web Text))

Max Activating Token Index: 749

Text #7

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.8680. Min Act: -0.0000

Data Index: 681684 (C4 (Web Text))

Max Activating Token Index: 514

Text #8

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.4957. Min Act: -0.0000

Data Index: 582207 (C4 (Web Text))

Max Activating Token Index: 818

Text #9

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.6360. Min Act: -0.0000

Data Index: 1120778 (C4 (Web Text))

Max Activating Token Index: 224

Text #10

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.7353. Min Act: -0.0000

Data Index: 1222421 (C4 (Web Text))

Max Activating Token Index: 428

Text #11

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.6606. Min Act: -0.0000

Data Index: 967876 (C4 (Web Text))

Max Activating Token Index: 240

Text #12

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.5494. Min Act: -0.0000

Data Index: 92128 (C4 (Web Text))

Max Activating Token Index: 154

Text #13

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.6356. Min Act: -0.0000

Data Index: 1124150 (C4 (Web Text))

Max Activating Token Index: 673

Text #14

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.3411. Min Act: -0.0000

Data Index: 270254 (C4 (Web Text))

Max Activating Token Index: 970

Text #15

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.5124. Min Act: -0.0000

Data Index: 1633529 (Python Code)

Max Activating Token Index: 1000

Text #16

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.5783. Min Act: -0.0000

Data Index: 416462 (C4 (Web Text))

Max Activating Token Index: 710

Text #17

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.4907. Min Act: -0.0000

Data Index: 285889 (C4 (Web Text))

Max Activating Token Index: 227

Text #18

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.5264. Min Act: -0.0000

Data Index: 501256 (C4 (Web Text))

Max Activating Token Index: 174

Text #19

Max Range: 2.8980. Min Range: -2.8980

Max Act: 1.4778. Min Act: -0.0000

Data Index: 528511 (C4 (Web Text))

Max Activating Token Index: 93