Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1090 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.8387. Min Act: -0.0000

Data Index: 1791670 (The Pile)

Max Activating Token Index: 908

Text #1

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.7556. Min Act: -0.0000

Data Index: 617279 (The Pile)

Max Activating Token Index: 963

Text #2

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.6529. Min Act: -0.0000

Data Index: 1144225 (The Pile)

Max Activating Token Index: 669

Text #3

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4996. Min Act: -0.0000

Data Index: 734475 (The Pile)

Max Activating Token Index: 580

Text #4

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.6917. Min Act: -0.0000

Data Index: 1658366 (The Pile)

Max Activating Token Index: 214

Text #5

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.5660. Min Act: -0.0000

Data Index: 386560 (The Pile)

Max Activating Token Index: 387

Text #6

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.6820. Min Act: -0.0000

Data Index: 1046080 (The Pile)

Max Activating Token Index: 101

Text #7

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.6224. Min Act: -0.0000

Data Index: 1876816 (The Pile)

Max Activating Token Index: 387

Text #8

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4815. Min Act: -0.0000

Data Index: 944865 (The Pile)

Max Activating Token Index: 70

Text #9

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.5273. Min Act: -0.0000

Data Index: 1446954 (The Pile)

Max Activating Token Index: 309

Text #10

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.6062. Min Act: -0.0000

Data Index: 1988051 (The Pile)

Max Activating Token Index: 258

Text #11

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.6076. Min Act: -0.0000

Data Index: 359543 (The Pile)

Max Activating Token Index: 939

Text #12

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4567. Min Act: -0.0000

Data Index: 1907035 (The Pile)

Max Activating Token Index: 747

Text #13

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.5367. Min Act: -0.0000

Data Index: 1927885 (The Pile)

Max Activating Token Index: 602

Text #14

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4486. Min Act: -0.0000

Data Index: 1606097 (The Pile)

Max Activating Token Index: 449

Text #15

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4769. Min Act: -0.0000

Data Index: 1832892 (The Pile)

Max Activating Token Index: 206

Text #16

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4439. Min Act: -0.0000

Data Index: 445252 (The Pile)

Max Activating Token Index: 804

Text #17

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.4487. Min Act: -0.0000

Data Index: 1342027 (The Pile)

Max Activating Token Index: 180

Text #18

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.5727. Min Act: -0.0000

Data Index: 1242867 (The Pile)

Max Activating Token Index: 981

Text #19

Max Range: 1.8387. Min Range: -1.8387

Max Act: 1.2928. Min Act: -0.0000

Data Index: 928225 (The Pile)

Max Activating Token Index: 650