Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1716 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.8606. Min Act: -0.0000

Data Index: 1078442 (C4 (Web Text))

Max Activating Token Index: 931

Text #1

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.8799. Min Act: -0.0000

Data Index: 272735 (C4 (Web Text))

Max Activating Token Index: 95

Text #2

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.8680. Min Act: -0.0000

Data Index: 591097 (C4 (Web Text))

Max Activating Token Index: 735

Text #3

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.8415. Min Act: -0.0000

Data Index: 202079 (C4 (Web Text))

Max Activating Token Index: 465

Text #4

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7739. Min Act: -0.0000

Data Index: 118831 (C4 (Web Text))

Max Activating Token Index: 685

Text #5

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7614. Min Act: -0.0000

Data Index: 103825 (C4 (Web Text))

Max Activating Token Index: 757

Text #6

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.8078. Min Act: -0.0001

Data Index: 145399 (C4 (Web Text))

Max Activating Token Index: 708

Text #7

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7438. Min Act: -0.0000

Data Index: 233246 (C4 (Web Text))

Max Activating Token Index: 624

Text #8

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7646. Min Act: -0.0000

Data Index: 938997 (C4 (Web Text))

Max Activating Token Index: 150

Text #9

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7353. Min Act: -0.0000

Data Index: 812334 (C4 (Web Text))

Max Activating Token Index: 514

Text #10

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7238. Min Act: -0.0001

Data Index: 202858 (C4 (Web Text))

Max Activating Token Index: 971

Text #11

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6722. Min Act: -0.0000

Data Index: 32382 (C4 (Web Text))

Max Activating Token Index: 194

Text #12

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6891. Min Act: -0.0000

Data Index: 409437 (C4 (Web Text))

Max Activating Token Index: 59

Text #13

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6568. Min Act: -0.0000

Data Index: 300056 (C4 (Web Text))

Max Activating Token Index: 257

Text #14

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6507. Min Act: -0.0000

Data Index: 824410 (C4 (Web Text))

Max Activating Token Index: 162

Text #15

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6214. Min Act: -0.0001

Data Index: 720594 (C4 (Web Text))

Max Activating Token Index: 23

Text #16

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.7082. Min Act: -0.0000

Data Index: 1076327 (C4 (Web Text))

Max Activating Token Index: 739

Text #17

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6370. Min Act: -0.0001

Data Index: 185550 (C4 (Web Text))

Max Activating Token Index: 296

Text #18

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6228. Min Act: -0.0000

Data Index: 956086 (C4 (Web Text))

Max Activating Token Index: 658

Text #19

Max Range: 0.8799. Min Range: -0.8799

Max Act: 0.6233. Min Act: -0.0001

Data Index: 642869 (C4 (Web Text))

Max Activating Token Index: 424