Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 154 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.3571. Min Act: -0.0000

Data Index: 449823 (The Pile)

Max Activating Token Index: 1016

Text #1

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.3853. Min Act: -0.0000

Data Index: 229464 (The Pile)

Max Activating Token Index: 411

Text #2

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.2304. Min Act: 0.0000

Data Index: 1514876 (The Pile)

Max Activating Token Index: 958

Text #3

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.2176. Min Act: -0.0000

Data Index: 745572 (The Pile)

Max Activating Token Index: 781

Text #4

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1977. Min Act: -0.0000

Data Index: 162621 (The Pile)

Max Activating Token Index: 824

Text #5

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1312. Min Act: -0.0000

Data Index: 750536 (The Pile)

Max Activating Token Index: 104

Text #6

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1385. Min Act: -0.0000

Data Index: 92877 (The Pile)

Max Activating Token Index: 131

Text #7

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1773. Min Act: -0.0000

Data Index: 1530761 (The Pile)

Max Activating Token Index: 516

Text #8

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.2105. Min Act: -0.0000

Data Index: 571867 (The Pile)

Max Activating Token Index: 736

Text #9

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1577. Min Act: -0.0000

Data Index: 1785459 (The Pile)

Max Activating Token Index: 972

Text #10

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1455. Min Act: -0.0000

Data Index: 1075369 (The Pile)

Max Activating Token Index: 583

Text #11

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1953. Min Act: -0.0000

Data Index: 1576198 (The Pile)

Max Activating Token Index: 317

Text #12

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1025. Min Act: -0.0000

Data Index: 1749182 (The Pile)

Max Activating Token Index: 479

Text #13

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1130. Min Act: -0.0000

Data Index: 314599 (The Pile)

Max Activating Token Index: 373

Text #14

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1861. Min Act: -0.0000

Data Index: 1537375 (The Pile)

Max Activating Token Index: 561

Text #15

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.0904. Min Act: -0.0000

Data Index: 628968 (The Pile)

Max Activating Token Index: 894

Text #16

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.0358. Min Act: 0.0000

Data Index: 290741 (The Pile)

Max Activating Token Index: 479

Text #17

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1641. Min Act: -0.0000

Data Index: 482913 (The Pile)

Max Activating Token Index: 310

Text #18

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.1472. Min Act: 0.0000

Data Index: 1874130 (The Pile)

Max Activating Token Index: 419

Text #19

Max Range: 1.3853. Min Range: -1.3853

Max Act: 1.0042. Min Act: -0.0000

Data Index: 1415605 (The Pile)

Max Activating Token Index: 806