Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2552 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3939. Min Act: -0.0002

Data Index: 827290 (C4 (Web Text))

Max Activating Token Index: 475

Text #1

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3997. Min Act: -0.0002

Data Index: 1285104 (C4 (Web Text))

Max Activating Token Index: 842

Text #2

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3715. Min Act: -0.0002

Data Index: 515480 (C4 (Web Text))

Max Activating Token Index: 745

Text #3

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3835. Min Act: -0.0002

Data Index: 465050 (C4 (Web Text))

Max Activating Token Index: 745

Text #4

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3890. Min Act: -0.0002

Data Index: 715892 (C4 (Web Text))

Max Activating Token Index: 443

Text #5

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3745. Min Act: -0.0002

Data Index: 707691 (C4 (Web Text))

Max Activating Token Index: 446

Text #6

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3708. Min Act: -0.0002

Data Index: 1019712 (C4 (Web Text))

Max Activating Token Index: 252

Text #7

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3744. Min Act: -0.0002

Data Index: 1026948 (C4 (Web Text))

Max Activating Token Index: 106

Text #8

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3511. Min Act: -0.0002

Data Index: 794134 (C4 (Web Text))

Max Activating Token Index: 593

Text #9

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3623. Min Act: -0.0002

Data Index: 1078089 (C4 (Web Text))

Max Activating Token Index: 790

Text #10

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3477. Min Act: -0.0002

Data Index: 999901 (C4 (Web Text))

Max Activating Token Index: 756

Text #11

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3577. Min Act: -0.0002

Data Index: 87229 (C4 (Web Text))

Max Activating Token Index: 171

Text #12

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3535. Min Act: -0.0002

Data Index: 573030 (C4 (Web Text))

Max Activating Token Index: 229

Text #13

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3267. Min Act: -0.0002

Data Index: 85421 (C4 (Web Text))

Max Activating Token Index: 220

Text #14

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3200. Min Act: -0.0002

Data Index: 1280096 (C4 (Web Text))

Max Activating Token Index: 120

Text #15

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3151. Min Act: -0.0002

Data Index: 518306 (C4 (Web Text))

Max Activating Token Index: 819

Text #16

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3014. Min Act: -0.0002

Data Index: 273641 (C4 (Web Text))

Max Activating Token Index: 588

Text #17

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.3018. Min Act: -0.0002

Data Index: 826311 (C4 (Web Text))

Max Activating Token Index: 411

Text #18

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.2838. Min Act: -0.0002

Data Index: 344327 (C4 (Web Text))

Max Activating Token Index: 257

Text #19

Max Range: 0.3997. Min Range: -0.3997

Max Act: 0.2785. Min Act: -0.0002

Data Index: 119231 (C4 (Web Text))

Max Activating Token Index: 453